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Abstract 


This  report  summarizes  the  results  of  a  literature  review  conducted  to  recommend  suitable 
adaptation  mechanisms  for  an  intelligent  tutoring  system  (ITS).  Intelligent  Tutoring  technologies 
have  been  identified  by  Defence  Research  &  Development  Canada  (DRDC)  -  Toronto  as  an 
effective  training  aid  for  the  Canadian  Forces  (CF).  A  specific  CF  training  course,  the 
Improvised  Explosive  Device  Disposal  (IEDD)  Operator  Course,  has  been  identified  as  a  target 
for  the  implementation  and  evaluation  of  intelligent  tutoring  technologies.  The  review  involved 
surveying  and  synthesizing  applicable  literature  to  develop  recommendations  and  best  practices 
pertaining  to  the  integration  of  these  technologies  into  an  ITS  for  the  IEDD  Operator  Course  to 
improve  questioning  skills.  The  recommended  four  adaptation  mechanisms  for  implementation 
into  the  IEDD  ITS  were  as  follows: 

•  Eye-tracking  technologies  to  support  attention  tracking; 

•  Cognitive  learning  styles  to  support  customization  of  the  learning  environment; 

•  Psychophysiological  indices  to  support  cognitive  state  motoring;  and 

•  Performance  measures  to  support  engagement  tracking. 
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Resume 


Le  rapport  presente  les  resultats  d’une  analyse  documentaire  visant  a  recommander  des 
mecanismes  d’adaptation  appropries  pour  un  systeme  tutoriel  intelligent  (STI).  Des  technologies 
de  tutorat  intelligent  ont  ete  determinees  par  Recherche  et  developpement  pour 
la  defense  Canada  (RDDC)  -  Toronto  en  tant  qu’outil  efficace  d’aide  a  l’instruction  des  Forces 
canadiennes  (FC).  Un  cours  particular  d’ instruction  des  FC,  le  Cours  d’operateur  de  la 
neutralisation  des  engins  explosifs  improvises  (IEDD),  a  ete  identifie  comme  convenant  a  la  mise 
en  oeuvre  et  a  revaluation  des  technologies  de  tutorat  intelligent.  L’ analyse  consistait  a  etudier  et 
a  resumer  les  documents  pertinents  afin  de  formuler  des  recommandations  ainsi  que  d’etablir  les 
meilleures  pratiques  relatives  a  l’integration  de  ces  technologies  en  un  systeme  tutoriel 
intelligent  pour  le  Cours  d’operateur  IEDD  afin  d’ameliorer  les  aptitudes  d’ interrogation. 
Les  quatre  mecanismes  d’adaptation  que  Ton  propose  de  mettre  en  oeuvre  dans  le  STI  IEDD  sont 
les  suivants  : 

•  technologies  de  suivi  du  regard  a  l’appui  du  suivi  de  1’ attention; 

•  styles  d’apprentissage  cognitif  a  l’appui  de  l’adaptation  de  l’environnement  d’apprentissage; 

•  indices  psychophysiologiques  a  l’appui  du  suivi  de  l’etat  cognitif; 

•  indie ateurs  de  rendement  a  l’appui  du  suivi  de  1’ engagement. 


11 


DRDC  Toronto  TR  2010-074 


Executive  summary 


Suitable  Adaptation  Mechanisms  for  Intelligent  Tutoring 
Technologies: 

Ming  Hou,  Suzanna  Sobieraj,  Chelsea  Kramer,  Jana  Lee  Tryan,  Simon  Banbury, 
and  Kristine  Osgoode;  DRDC  Toronto  TR  2010-074;  Defence  R&D  Canada  - 
Toronto;  December  2010. 

This  document  presents  the  results  of  a  literature  review  conducted  to  recommend  specific 
intelligent  tutoring  technologies  for  integration  into  a  Canadian  Forces  (CF)  training  course. 
Previous  research  efforts  have  identified  a  specific  CF  training  course,  the  Improvised  Explosive 
Device  Disposal  (IEDD)  Operator  Course  as  a  target  for  the  implementation  and  evaluation  of 
intelligent  tutoring  technologies.  Importantly,  in  the  IEDD  training  course  the  failure  rate  for  the 
team  leader  is  significantly  high  (40%).  This  failure  rate  has  been  attributed  to  deficiencies  in 
students’  threat  assessment  and  questioning  technique  skills. 

The  objective  of  this  review  was  to  identify  suitable  adaptation  mechanisms  that  could  be  used  to 
improve  the  effectiveness  of  an  intelligent  tutoring  system  (ITS),  which  will  be  implemented 
within  the  IEDD  Operator  Course.  The  ITS  can  customize  training  material  and  provide  real-time 
feedback  to  improve  students’  questioning  skills.  A  large  body  of  knowledge  was  surveyed  and 
synthesized  to  develop  recommendations  and  highlight  best  practices.  Scientific,  defence, 
government,  and  internet-based  sources  were  searched  for  literature  pertaining  to  principles  and 
best  practices  for  adaptation  mechanisms  in  an  ITS.  The  emphasis  of  the  search  was  to  find 
developing  and  maturing  technologies  (e.g.,  Commercial  Off-The-Shelf  (COTS)  products) 
instead  of  theoretical  or  conceptual  insights. 

The  results  of  the  literature  review  demonstrated  a  range  of  maturing  technologies  that  can  be 
applied  to  the  ITS  designed  to  augment  the  IEDD  Operator  Course.  The  recommended 
technologies  for  implementation  into  the  IEDD  ITS  are  as  follows: 

•  Eye-tracking  technologies  to  support  attention  tracking; 

•  Cognitive  learning  styles  to  support  customization  of  the  learning  environment; 

•  Psychophysiological  indices  to  support  monitoring  cognitive  states  (e.g.,  workload);  and 

•  Performance  measures  to  support  engagement  tracking. 
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Ce  document  presente  les  resultats  d’une  analyse  documentaire  visant  a  recommander  des 
technologies  particulieres  de  tutorat  intelligent  a  integrer  dans  un  cours  de  formation  des  Forces 
canadiennes  (FC).  Des  recherches  anterieures  ont  permis  de  determiner  un  cours  particulier 
des  FC,  soit  le  cours  d’operateur  de  la  neutralisation  des  engins  explosifs  improvises  (IEDD),  en 
tant  que  cible  pour  la  mise  en  oeuvre  et  revaluation  des  technologies  de  tutorat  intelligent.  Fait 
important,  dans  le  cours  de  formation  IEDD,  le  taux  d’echec  pour  le  chef  d’equipe  est  tres  eleve 
(40  p.  100).  Ce  taux  d’echec  a  ete  attribue  aux  competences  insuffisantes  des  stagiaires  en  matiere 
devaluation  des  menaces  et  d’ interrogation. 

L’ analyse  documentaire  avait  pour  objet  de  trouver  des  mecanismes  d’ adaptation  utiles  pouvant 
servir  a  ameliorer  l’efficacite  d’un  systeme  tutoriel  intelligent  qui  sera  mis  en  oeuvre  dans  le  cadre 
du  cours  d’operateur  IEDD.  Le  STI  peut  adapter  le  materiel  de  formation  et  offrir  une  retroaction 
en  temps  reel  afin  d’ameliorer  les  techniques  d’ interrogation  des  stagiaires.  Un  vaste  ensemble  de 
connaissances  a  ete  analyse  et  resume  afin  de  formuler  des  recommandations  et  de  souligner  les 
pratiques  exemplaires.  On  a  effectue  des  recherches  dans  des  sources  scientifiques,  de  la  defense, 
du  gouvernement  et  d’lnternet  afin  de  trouver  de  la  documentation  sur  les  principes  et  les 
pratiques  exemplaires  relatifs  aux  mecanismes  d’ adaptation  dans  un  STI.  Le  but  principal  de  la 
recherche  etait  de  trouver  des  technologies  en  voie  d’evolution  et  de  maturation  (p.  ex.,  des 
produits  logiciels  commerciaux)  et  non  des  considerations  theoriques  et  conceptuelles. 

Selon  les  resultats  de  1’ analyse  documentaire,  une  gamme  de  technologies  en  voie  de  maturation 
peuvent  etre  appliquees  au  STI  conqu  pour  completer  le  cours  d’operateur  IEDD.  Void  les 
technologies  que  l’on  recommande  d’integrer  au  systeme  tutoriel  intelligent  en  matiere  d’lEDD  : 

•  technologies  de  suivi  du  regard  a  l’appui  du  suivi  de  1’ attention; 

•  styles  d’apprentissage  cognitif  a  l’appui  de  l’adaptation  de  l’environnement  d’apprentissage; 

•  indices  psychophysiologiques  a  l’appui  du  suivi  des  etats  cognitifs  (p.  ex.,  la  charge  de 
travail); 

•  indie ateurs  de  rendement  a  l’appui  du  suivi  de  1’ engagement. 
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1  Introduction 


This  document  presents  the  results  of  a  literature  review  conducted  to  recommend  specific 
intelligent  tutoring  technologies  for  integration  into  Department  of  National  Defence  (DND) 
learning  environments  in  order  to  improve  Canadian  Forces  (CF)  distance  learning  capabilities. 
Specifically,  a  CF  training  course,  the  Improvised  Explosive  Device  Disposal  (IEDD)  Operator 
Course,  has  been  identified  as  an  ideal  target  for  the  implementation,  evaluation,  and 
demonstration  of  intelligent  tutoring  technologies. 

1.1  Background 

The  IEDD  Operator  Course  teaches  CF  personnel  to  identify,  disrupt,  and  dispose  of  intelligent 
explosive  devices  (IEDs).  Students  are  also  taught  how  to  identify,  recognize,  and  formulate  an 
accurate  threat  assessment  of  the  suspected  IED,  and  to  provide  advice  on  immediate  protective 
measures  against  hazards  associated  with  chemical,  biological,  and  radiological  (CBR) 
improvised  devices. 

At  present,  the  course  is  delivered  using  a  combination  of  classroom  and  field-based  training.  One 
of  the  biggest  difficulties  associated  with  the  course  is  gathering  the  students  together  for  the 
classroom  and  scenario-based  training.  Questioning  technique  is  taught  in  a  classroom  setting 
using  a  series  of  text-based  PowerPoint  slides,  which  essentially  provide  definitions  of  the 
different  types  of  questions  that  one  could  ask  (e.g.,  open-ended,  leading,  etc.),  and  when  each 
may  or  may  not  be  appropriate.  Misinterpretation  of  pertinent  cues  within  the  evaluation 
scenarios  can  mislead  students,  causing  them  to  identify  an  incorrect  IED  Render  Safe  Procedure 
(RSP)  and,  ultimately,  leading  them  to  fail  the  course. 

Currently,  the  course  failure  rate  for  the  IEDD  team  leader  is  significantly  high  (40%).  This 
failure  rate  has  been  attributed  to  difficulties  during  situation  assessment  and  decision-making 
biases  related  to  working  under  considerable  time  pressure  and  stress.  Examples  of  decision¬ 
making  biases  include  confirmation  bias  and  belief  perseverance.  The  former  is  a  tendency  to 
seek  data  that  confirms  one’s  beliefs  (Wason,  1960),  while  the  latter  is  a  resistance  to  changing 
one’s  beliefs  in  the  face  of  disconfirming  evidence  (Ross,  Lepper,  &  Hubbard,  1975). 

Decision-making  biases  expressly  affect  the  portion  of  the  IED  identification  process  that 
involves  both  the  close  inspection  of  the  device  itself,  and  the  gathering  of  information  about  the 
device.  This  information  is  achieved  through  questioning  third  parties  (e.g.,  civilians,  local  police, 
CF  personnel,  etc.)  to  determine  the  level  of  threat  posed  to  personnel  by  the  device.  Therefore, 
the  implementation  of  intelligent  tutoring  technologies  to  improve  decision-making  processes  and 
situation  assessment  has  been  targeted  explicitly  at  the  IED  identification  process. 

To  address  the  IEDD  high  failure  rate  training  problem  and  to  improve  trainees’  situation 
assessment  and  decision-making  skills,  Defence  Research  and  Development  Canada  (DRDC)  - 
Toronto  has  identified  intelligent  tutoring  technologies  as  a  solution. 
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Intelligent  tutoring  technologies  try  to  minimize  the  mismatch  between  learner  needs  and  the 
learning  environment  as  they  attempt  to  elicit  learner  needs  via  performance  measures  and 
explicit  modeling  practices.  However,  these  systems  tend  to  ignore  learner  differences 
(Karamouzis,  2006).  The  key  challenge  addressed  in  this  report  is  the  ability  to  create  an  ITS  that 
can  customize  the  learning  experience  to  the  individual  and  to  incorporate  his  or  her  differences. 

An  Intelligent  Tutoring  System  (ITS)  broadly  encapsulates  any  self -regulating  computer  program 
that  contains  some  intelligence  and  can  be  used  for  the  control,  delivery,  and  assessment  of 
learning  content.  An  ITS  is  the  environment  in  which  adaptation  for  the  learner  is  accomplished. 
Thus,  complex  algorithms  are  designed  to  rely  on  feedback  from  the  learner’s  performance,  prior 
exposure  to  knowledge,  and  learning  rate  to  deliver,  evaluate,  and  react  according  to  pedagogical 
principles,  educational  goals,  and  implementation  tools. 

The  goal  of  most  ITSs  is  to  provide  the  learner  with  the  same  benefits  as  one-on-one  instruction, 
automatically.  While  tutoring  between  an  expert  and  a  novice  has  shown  to  be  a  very  effective 
way  to  learn,  computer-based,  unsupervised  instruction  is  vulnerable  to  student  disengagement, 
boredom,  frustration,  and  an  overall  lack  of  effective  learning  (Woodill,  2004).  To  remedy  such 
deterrents,  intelligent  tutoring  mechanisms  must  customize  the  content,  format,  modality  and  time 
of  content  exposure  to  the  individual’s  learning  style  and  preferences.  Intelligent  tutoring  systems 
offer  one  solution  to  help  to  achieve  these  goals. 

In  the  context  of  IEDD  Operator  training,  the  challenge  is  for  the  adaptation  mechanisms  in  an 
ITS  to  improve  trainees’  identification  of  IEDs,  their  questioning  techniques  and  threat 
assessment  skills.  To  address  this  issue,  DRDC  Toronto  has  reviewed  a  wide  range  of  possible 
intelligent  tutoring  technologies  and  has  identified  four  adaptation  mechanisms  suitable  for 
implementation  in  an  ITS.  They  are  eye-tracking  technologies,  cognitive  learning  styles, 
psychophysiological  indices,  and  performance/attention  tracking.  However,  details  of  these 
adaptation  mechanisms  still  need  to  be  reviewed  in  terms  of  their  theoretical  foundation, 
validation,  best  practices,  and  their  feasibility  for  the  ITS  (Banbury,  Osgoode,  Unrau,  & 
Kramer,  2009). 

1.2  Objectives 

The  objective  of  this  review  was  to  identify  suitable  adaptation  mechanisms  that  could  be  used  to 
improve  the  effectiveness  of  an  ITS  that  will  be  developed  for  the  IEDD  Operator  Course.  This 
objective  was  achieved  by  combining  a  sequence  of  related  reviews  with  current  and  more 
detailed  information  on  purchasable  technology,  in  order  to  provide  specific  and  concrete 
recommendations  on  the  products  that  should  be  purchased  and  implemented  into  the  IEDD 
Operator  Course  ITS.  The  work  described  in  this  report  reviews  the  literature  pertaining  to 
principles  and  best  practices  for  intelligent  tutoring  technologies  as  pertaining  to  operator 
cognitive  state  (e.g.,  workload,  stress),  learning  style,  engagement  and  performance.  The  results 
of  this  review  were  used  to  provide  recommendations  and  guidelines  for  the  integration  of  these 
technologies  into  the  IEDD  ITS. 
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1 .3  Scope 


The  structure  of  this  document  is  described  below: 

•  Section  1.  Presents  a  brief  introduction  to  the  IEDD  Operator  Course,  and  an  overview  of  the 

current  research; 

•  Section  2.  Presents  a  detailed  description  of  the  review  method  used  for  the  current  research; 

•  Section  3.  Presents  the  results  of  the  survey  and  review  of  eye-tracking  technologies  and 
provides  recommendations  for  the  implementation  of  these  technologies  within  the  realm  of 
the  proposed  IEDD  ITS; 

•  Section  4.  Presents  the  results  of  the  survey  and  review  of  cognitive  learning  styles  and 
provides  recommendations  for  the  implementation  of  these  technologies  within  the  realm  of 
the  proposed  IEDD  ITS ; 

•  Section  5.  Presents  the  results  of  the  survey  and  review  of  psychophysiological  indices  and 
provides  recommendations  for  the  implementation  of  these  technologies  within  the  realm  of 
the  proposed  IEDD  ITS; 

•  Section  6.  Presents  the  results  of  the  survey  and  review  of  performance  measures  and 
attention  tracking,  and  provides  recommendations  for  the  implementation  of  these 
technologies  within  the  realm  of  the  proposed  IEDD  ITS ; 

•  Section  7.  Presents  overall  conclusions,  purchasing  recommendations,  suggestions  for  future 

work,  and  implementation  plans  with  respect  to  intelligent  tutoring. 
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2  Review  Method 


This  section  presents  the  process  for  a  review  of  intelligent  tutoring  technologies  which  led  to 
recommendations  and  suggested  uses  for  their  implementation  to  improve  CF  distance 
learning  capabilities.  All  recommendations  and  best  practices  are  geared  towards  the  IEDD 
Operator  Course. 

This  focus  of  this  report  includes  the  following  four  adaptation  mechanisms  and  technologies: 

•  Eye-tracking  technologies; 

•  Cognitive  learning  styles; 

•  Psychophysiological  indices;  and 

•  Performance  measures  and  attention  tracking. 

It  should  be  noted  that  the  objective  of  each  review  section  was  to  examine  studies  that  have  been 
successful  (or  at  least  show  potential)  at  implementing  the  target  adaptation  mechanism  or 
technology.  Fulfilling  this  objective  was  an  essential  precursor  to  identifying  key  factors  in  the 
implementation  and  evaluation  of  each  adaptation  mechanism  or  technology  within  an  ITS  for  the 
IEDD  Operator  Course. 

2.1  Literature  Search  Process  and  Considerations 

Scientific,  defence  [e.g.,  United  States  (US)  defence  and  North  Atlantic  Treaty  Organization 
reports  (NATO)],  government  (e.g.,  DRDC  archives)  and  internet-based  sources  were  searched 
for  literature  pertaining  to  the  four  previously  mentioned  adaptation  mechanisms  and 
technologies.  Lone-standing  technology  products  were  included  to  provide  greater  breadth  and 
depth  to  the  range  of  technologies  used,  as  journal  articles  typically  feature  only  the  same  few 
mainstream  brands.  Typically,  a  university  or  institution  (authoring  the  articles)  will  purchase  a 
sole  piece  of  premium  technology  which  is  then  used  by  many  people.  As  purchasing  the  most 
expensive  type  of  equipment  may  not  be  desirable  for  the  current  project,  it  was  important  to 
include  a  range  of  options  within  each  of  the  adaptation  techniques  discussed. 

The  emphasis  of  the  search  was  finding  developing  and  maturing  technologies  [e.g.,  Commercial- 
Off-The-Shelf  (COTS)  products]  to  measure  these  characteristics,  instead  of 
theoretical/conceptual  insights.  As  such,  the  review  highlights  technologies  that  could  potentially 
be  used,  or  that  were  used  successfully  with  other  types  of  ITSs. 

A  further  consideration  of  the  review  was  the  ITS  context  for  which  this  research  will  be 
developed.  The  ultimate  goal  is  to  create  an  ITS  to  enhance  the  learning  environment  and 
ultimately  improve  training  for  the  IEDD  Operator  Course.  The  current  high  failure  rate  (i.e., 
40%)  is  believed  to  stem  from  problems  in  teaching  and  learning  questioning  techniques.  As  a 
result,  the  type  of  technology  chosen  must  cater  to  training  operators  within  this  context. 
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The  task  of  choosing  products  based  on  quality  or  validated  research  is  further  complicated  by 
other  constraining  factors  that  will  influence  the  type  of  technology  actually  used.  In  other  words, 
products  that  are  feasible  for  the  project  may  differ  from  the  “best-available”  product  in  each 
category.  The  multiple  training  environment  factors  that  were  considered  when  conducting  the 
literature  search  for  technologies  are  presented  in  Table  1. 


Table  1.  Training  environment  factors  and  ITS  implementation  considerations. 


Training  Environment 

ITS  Implementation  Considerations 

1.  Will  the  user  operate  the  ITS  from  a  laptop 
or  personal  computer  (PC),  or  is  the 
training  immersive  in  theatre? 

>  Need  to  consider  wireless  options, 

mobility,  fidelity  of  virtual  environment, 
visibility,  lighting 

2.  Will  training  occur  in  a  classroom  setting 
with  other  students,  or  work  individually? 

>  Need  to  consider  distractions,  noise-level, 
attention,  and  movement 

3.  How  many  mechanisms  will  be  used  at 
once? 

>  Need  to  consider  if  technologies  interfere 
with  one  another? 

4.  Will  students  work  remotely,  as  a  distance 
learner  with  limited  support? 

>  Tutor  will  need  to  be  self-sufficient  -  i.e., 
able  to  run  without  supervision  of  the 
researcher  or  other  expert 

Considering  that  the  current  study  will  start  with  threat  assessment  and  questioning  technique, 
which  is  only  a  small  subset  of  the  course  content,  this  review  also  considered  mechanisms  that 
could  benefit  the  course  beyond  the  ITS  itself.  For  instance,  reviewed  in  detail  are  suggestions  on 
how  training  introspective  awareness  of  differences  between  one’s  own  learning  style  from  others 
can  improve  learning  in  a  classroom  setting.  The  following  section  describes  the  process  of 
detailed  review  for  each  reference  on  adaptive  mechanisms. 

2.2  Review  Summary  on  Adaptation  Mechanisms 

The  following  sections  detail  the  most  relevant  references  and  technologies  pertaining  to  eye 
tracking,  cognitive  learning  styles,  psychophysiological  measures,  and  attention/performance 
tracking  as  means  of  implementing  an  ITS.  Where  relevant,  references  state  the  general  type  of 
product  (e.g.,  headmounted  eye-tracker),  brand  name  (e.g.,  Tobii™)  and  software  when  known 
(e.g.,  MatLab™)  that  were  used  in  the  study.  The  goal  was  to  emphasize  the  technologies  actually 
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used  in  published  studies.  Furthermore,  similar  technologies  that  were  not  used  within  the  context 
of  a  published  article,  but  were  deemed  relevant  to  the  current  literature  review  were 
included  in  order  to  provide  more  depth  (i.e.,  “comparison  shopping”)  into  the  overall  search  of 
adaptive  mechanisms. 

Table  2  shows  the  categorization  of  the  73  references  included  in  this  review.  The  number  of 
articles  is  allocated  to  each  of  the  four  literature  sections:  Eye-tracking;  Cognitive  Learning 
Styles;  Psychophysiological  Measures;  and  Performance  Tracking.  Each  literature  section  begins 
with  a  high-level  description  of  the  content  of  the  literature  and  products  that  were  reviewed, 
followed  by  a  list  of  the  references  examined,  and  a  table  that  summarizes  the  characteristics  of 
the  adaptive  mechanisms  within  the  context  implementation  for  the  IEDD  Operator  Course.  This 
summary  table  is  used  to  infer  recommendations  for  the  implementation  of  these  technologies 
into  the  IEDD  Course. 


Table  2.  Number  of  references  grouped  by  literature  review  topic  area 


Literature  Review  Topic  Area 

Eye-Tracking 

Learning  Styles 

Psychophysiological 

Performance  Tracking 

Total  Number 
of  References 

19 

10 

30 

14 

Table  3  describes  the  number  of  articles  classified  in  terms  of  the  level  of  experimentation 
involved  (i.e.,  conceptual  study  involving  no  evaluation,  single  laboratory-based  evaluation, 
single  simulator  or  field-based  evaluation,  and  multiple  laboratory-,  field-  or  simulator-based 
evaluations),  degree  of  peer  review  (i.e.,  none,  conference  proceedings  and/or  journal  article),  and 
proximity  and  relevance  to  military  domains  (i.e.,  basic,  business,  industrial  and  military). 


Table  3.  Number  of  references  grouped  by  level  of  experimentation,  peer  review  and 

domain  relevance 


Level  of 

Experimentation 

Peer  Review 

Domain  Relevance 

Stand-Alone 

Products 

Conceptual 

Single  Lab 
Evaluation 

Single 

Simulation  or 
Field 

Multiple 

Evaluations 

None 

Conference 

Journal 

Academic 

Business 

Industrial 

Military 

COTS 

Total 

Number  of 
References 

6 

8 

17 

12 

3 

17 

23 

26 

1 

6 

9 

21 

6 


DRDC  Toronto  TR  2010-074 


The  statistics  shown  in  the  preceding  two  tables  demonstrate  that: 

•  The  breadth  of  articles  reviewed  is  sufficient;  a  large  number  of  articles  have  been  used  in  all 
four  topic  areas. 

•  The  articles  are  mostly  single  field/simulation  (23%)  or  multiple-based  studies  (16%); 

•  A  significant  proportion  of  articles  are  peer  reviewed  (34%);  and, 

•  A  significant  proportion  of  the  references  originate  from  the  academic  domain  (37%),  while 
29%  are  references  for  “stand-alone”  products  that  are  COTS 
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3  Eye-Tracking 


This  section  provides  a  brief  summary  on  processes,  uses,  and  types  of  eye-tracking  technology. 
This  review  is  intended  to  aid  the  understanding  of  the  reader,  but  should  not  be  considered  a 
comprehensive  description  of  all  eye-tracking  literature. 

3.1  Eye-Tracking  Processes 

Eye-tracking  is  the  process  of  measuring  either  the  point  of  gaze  ("where  we  are  looking")  or  the 
motion  of  an  eye  relative  to  the  head.  The  basic  assumption  of  eye-tracking  data  is  that  where  the 
subject  is  looking  is  what  they  are  attending  to.  While  this  report  acknowledges  the 
concept  of  “inattentional  blindness ”  -  the  phenomenon  of  not  perceiving  a  stimulus  in  plain  sight 
(Simons  &  Chabris,  1999)  -  further  separation  of  eye-tracking  from  perception  is  beyond  the 
scope  of  this  review. 

According  to  Viviani  (1990),  eye  tracking  can  be  used  to  gain  a  better  understanding  of  how  a 
visual  stimulus  is  being  encoded.  Eye  movement  is  typically  divided  into  fixations  (i.e.,  when  the 
eye  gaze  pauses  in  a  certain  position)  and  saccades  (i.e.,  when  the  eye  moves  to  another  position). 
The  resulting  series  of  fixations  and  saccades  is  called  a  scanpath  (Cassin  &  Solomon,  1990). 
Perception  is  considered  to  occur  while  the  eye  is  fixated,  as  opposed  to  during  a 
saccade  (e.g.,  North,  William,  Hodges,  Ward,  &  Ericsson,  2009).  Essentially,  this  is  based  on  the 
premise  underlying  the  eye  -  mind  hypothesis,  which  states  that  people  look  at  the  object  or 
location  that  they  are  thinking  about  (Just  &  Carpenter,  1988). 

For  instance,  if  an  individual  is  looking  at  a  particular  word,  it  is  assumed  that  they  are  processing 
(i.e.,  reading)  that  word.  In  research  involving  computer  interfaces,  based  on  the  eye  -  mind 
hypothesis,  we  are  able  to  determine  and  uncover  the  most  informative  areas  of  a  display  or 
scene.  As  such,  we  are  able  to  infer  what  is  being  processed  by  computing  eye  metrics  such  as 
dwell  time  within  a  defined  Area  of  Interest  (AOI)  (Rayner,  1998).  In  addition  to  dwell  time,  the 
sequential  nature  of  eye-movement  behaviour  has  been  linked  to  intent.  Brandt  and  Stark  (1997) 
found  evidence  supporting  sequential  processing  (scanpath)  where  they  concluded,  that  an 
“internalized,  cognitive  perceptual  model  must  be  in  control  of  these  scanpaths”  (p.  32).  In 
essence,  uncovering  the  sequence  of  eye-movements  from  one  area  of  interest  to  another  is 
valuable  in  understanding  the  strategies  and  planning  used  by  individuals  when  performing  a  task, 
which  are  often  shown  to  differ  among  expert  and  novices  in  a  given  area. 

The  central  one  or  two  degrees  of  the  visual  angle  (the  fovea)  provide  the  bulk  of  visual 
information;  the  input  from  larger  eccentricities  (the  periphery)  is  less  informative.  Hence,  the 
locations  of  fixations  along  a  scanpath  show  what  information  loci  on  the  stimulus  were 
processed  during  an  eye-tracking  session.  On  average,  fixations  last  for  around  200  ms  during  the 
reading  of  linguistic  text,  and  350  ms  during  the  viewing  of  a  scene.  Preparing  a  saccade  towards 
a  new  goal  takes  around  200  ms  (Cassin  &  Solomon,  1990). 

Scanpaths  are  useful  for  analyzing  cognitive  intent,  interest,  and  salience.  Other  biological  factors 
(e.g.,  gender)  may  also  affect  the  scanpath.  There  are  a  number  of  methods  for  measuring  eye 
movements  such  as  scan  paths,  but  the  most  commonly  used  method  in  research  is  the  eye-tracker 
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device.  The  most  popular  option  uses  video  images  from  which  the  eye  position  is  extracted, 
while  other  methods  are  based  on  the  electrooculogram  (EOG);  a  technique  for  measuring  the 
resting  potential  of  the  retina  which  results  in  a  record  of  eye  movements  (e.g.,  Bulling,  Roggen, 
&  Troster,  2008).  The  following  is  a  brief  summary  on  popular  types  of  eye-trackers  used  in  the 
literature  that  was  examined  in  this  review. 

3.2  Types  of  Eye-Trackers 

The  most  widely  used  current  designs  are  video-based  eye-trackers.  In  these  types  of  eye-trackers, 
a  camera  focuses  on  one  or  both  eyes  and  records  their  movement  as  the  viewer  looks  at  a 
stimulus.  Most  modern  eye-trackers  use  contrast  to  locate  the  centre  of  the  pupil,  in  addition  to 
infrared  and  near-infrared  non-collimated  light  to  create  a  corneal  reflection  (CR).  The  vector 
between  these  two  features  can  be  used  to  compute  gaze  intersection  with  a  surface  after  a  simple 
calibration  for  an  individual. 

Eye-tracking  setups  vary  greatly.  Some  eye-trackers  are  head-mounted,  some  require  the  head  to 
be  stable  (e.g.,  with  a  chin  rest),  and  some  function  remotely  and  automatically  to  track  the  head 
during  motion.  Most  use  a  sampling  rate  of  at  least  30  Hz.  Although  50/60  Hz  is  most  common, 
many  modern  video-based  eye-trackers  run  at  240,  350  or  even  1000/1250  Hz,  which  is 
needed  in  order  to  capture  the  detail  of  the  very  rapid  eye  movements  during  reading,  or  during 
studies  of  neurology. 

Eye-trackers  measure  the  rotation  of  the  eye  with  respect  to  the  measuring  system.  If  the 
measuring  system  is  head  mounted,  as  with  the  EOG,  then  eye-in-head  angles  are  measured.  If 
the  measuring  system  is  table  mounted,  as  with  scleral  search  coils  or  table  mounted  camera 
(“remote”)  systems,  then  gaze  angles  are  measured.  In  many  applications,  the  head  position  is 
fixed  using  a  bite  bar  (e.g.,  San  Agustin,  Skovsgaard,  Hansen,  &  Hansen,  2009),  a  forehead 
support  or  something  similar,  so  that  eye  position  and  gaze  are  the  same.  In  other  cases,  the  head 
is  free  to  move  (e.g.,  Tobii  Eye-Tracker),  and  head  movement  is  measured  with  systems  such  as 
magnetic-  or  video-  based  head  trackers. 

Headmounted  eye-trackers  are  fixed  on  top  of  the  user’s  heard  (e.g.,  Eyelink),  and  head  position 
and  direction  are  added  to  eye-in-head  direction  to  determine  gaze  direction.  Desktop  systems 
(e.g.,  EasyGaze)  are  tetherless  and  capture  eye  movement  remotely  in  front  of  the  participant. 
These  types  use  search  coils,  and  head  direction  is  subtracted  from  gaze  direction  to  determine 
eye-in-head  position. 

In  general,  eye-trackers  are  more  accurate  and  will  require  less  calibration  when  the  head  remains 
fixated,  allowing  only  the  eyes  to  move.  As  such,  remote  desktop  mountings  with  chin  rest 
options  may  provide  the  most  granular  measures,  but  may  be  uncomfortable  and  un-naturalistic 
for  wearers.  In  addition,  headmounted  eye-trackers  allow  for  more  movement,  but  can  be 
obtrusive  and  timely  to  set  up.  A  third  kind  of  eye-tracker  that  is  rapidly  gaining  popularity  is  the 
goggle  style  eye-tracker  (e.g.,  Mobile).  These  eye-trackers  are  often  wireless  using  Bluetooth 
technology,  and  are  designed  for  eye-tracking  in  real-life  environments  such  as  sport  research 
(e.g.,  Crews  &  Lutz,  2008). 
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3.3  Eye-Tracker  Use 


Eye-tracking  data,  specifically  eye  movements  or  scan  paths,  have  been  used  retrospectively  to 
analyze  usability  issues  and  human  performance  within  the  human  -  computer  interaction  domain 
(Jacob  &  Karn,  2003).  Real-time  eye-gaze  data  have  been  investigated  as  an  input  tool  for 
interface  interaction.  For  example,  Hornof,  Cavender,  and  Hoselton  (2004)  describe  a  system  that 
enabled  children  with  severe  motor  impairments  to  draw  pictures  by  just  moving  their  eyes.  A 
similar  eye-drawing  open-source  product  is  developed  by  EyeWriter  (EyeWriter  Initative:  New 
York,  NY),  whose  website  provides  detailed  instructions  on  how  to  create  your  own  drawing  tool 
using  parts  from  a  Playstation  gaming  system. 

Another  use  of  eye-tracking,  and  of  particular  use  and  interest  to  the  current  review,  is  real-time 
processing  of  a  user’s  gaze  to  interpret  user  non-explicit  cognitive  behaviours  for  online 
interaction  adaptation.  For  example,  Sibert,  Gokturk,  and  Lavine  (2000)  described  a  system  that 
tracks  the  reader’s  eye  movements  and,  using  principles  derived  from  reading  research,  aids  the 
reader  by  pronouncing  words  that  appear  to  be  hard  to  recognize. 

Bulling  et  al.  (2008)  propose  that  the  EOG  may  be  used  as  a  novel  measurement  technique  for 
wearable  eye-tracking  and  recognition  of  user  activity  and  attention  in  more  mobile  settings 
(Bulling  et  al.,  2008,  p.  1).  With  the  EOG,  electrodes  are  typically  placed  around  the  eye  so  that 
when  the  eye  moves  from  the  center  position  towards  one  electrode,  this  electrode  “sees”  the 
positive  side  of  the  retina  and  the  opposite  electrode  “sees”  the  negative  side  of  the  retina. 
Consequently,  a  potential  difference  occurs  between  the  electrodes.  Assuming  that  the  resting 
potential  is  constant,  the  recorded  potential  is  a  measure  for  the  eye  position.  A  major 
benefit  of  the  EOG  lies  in  the  minimal  amount  of  power  and  computation  that  is  required  for 
signal  processing. 

3.4  Results  of  Literature  Search  on  Eye-Tracking 
Technologies 

The  literature  search  on  this  subject  yielded  information  containing  eye-tracking  technology  that 
either  directly  involved,  or  was  considered  useful  for  implementation  into  intelligent  tutoring 
systems.  Table  4  presents  a  list  of  the  references  examined  in  this  literature  review,  classified  by 
type  of  eye-tracking  device  used  (e.g.,  headmounted),  and  by  product  specific  information. 
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Table  4.  Eye-tracking  technology  literature  search:  Reference  list 


Style 

Product 

Reference 

Head  Mounted  &  Goggle 

ASL  501/504 

•  Louwerse,  Graesser,  McNamara,  &  Lu  (2008) 

•  North  et  al.  (2009) 

SR  Research  Eyelink  II 

•  SR  Research  (2010) 

•  Zotov  et  al.  (2009) 

•  Keillor  et  al.  (2006/2007) 

EOG  Goggles 

•  Bulling  et  al.  (2008) 

Scene  Monocular 

•  Arrington  Research  (2010) 

ASL  Mobile  Eye 

•  Applied  Science  Laboratories 

Deja  View 

•  Li  &  Parkhurst  (2006) 

Eye  Writer 

•  Eye  writer  (2010) 

Desktop 

ASL  Eyetrac  6 

•  Applied  Science  Laboratories 

Design  Interactive  Easy  Gaze 

•  Design  Interactive  (2010) 

Tobii  (all) 

•  Klingner,  Kumar,  &  Hanrahan  (2008) 

•  Tobii  (2010) 

SR  Research  Eyelink  1000 

•  SR  Research  (2010) 

Gaze  Tracker 

•  San  Agustin  et  al.  (2009) 

LC  Technology  Easy  Gaze 

•  Sibert  et  al.  (2000) 

In  addition,  a  summary  of  the  desired  qualities  for  eye-tracking  technologies  are  presented  in 
Table  5.  This  table  was  based  on  a  list  of  requirements  designed  to  aid  in  the  selection  of  the  most 
appropriate  product  to  implement,  followed  by  recommendations  and  suggested  uses  for  the 
implementation  of  the  adaptive  e-learning  framework  of  the  IEDD  course,  based  on  the  literature 
reviewed.  The  technologies  within  the  eye-tracking  literature  were  evaluated  based  on  the 
following  criteria: 

•  COTS:  Is  this  product  currently  available  commercially,  off-the-shelf? 

•  Unobtrusive:  Is  this  product  invasive,  bulky,  or  in  any  way  uncomfortable  so  that  it  may 
affect  desire  to  wear.  Also,  is  it  difficult  to  setup  and  calibrate  alone? 

•  Affordable:  Is  this  product  within  the  reasonable  budget  to  spend? 

•  User  Support:  Does  the  product  offer  any  form  of  technical  support  or  training  after 
initial  purchase? 

•  Build  Own:  Does  the  product  offer  instructions  on  how  to  build  it  oneself? 

•  Leave  for  IEDD  course:  Can  this  product  remain  in  Canadian  Forces  Base  (CFB) 

Gagetown,  New  Brunswick  for  use  by  the  IEDD  course  students  when  the 
project  is  completed? 
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Table  5.  Summary  of  Eye-Tracker  Requirements  Checklist 


Style 

Product 

COTS 

Affordable 

Unobtrusive 

User 

Support 

Build 

Own 

Leave  for 
IEDD 
Course 

Head  Mounted  &  Goggle 

ASL  501/504 
(Applied  Science 
Laboratories:  Bedford, 
MA) 

yes 

no 

no 

yes 

no 

no 

Eyelink  II  (SR 
Research:  Kanata, 
Canada) 

yes 

no 

no 

yes 

no 

no 

EOG  Goggles  (Swiss 
Federal  Institute  of 
Technology:  Zurich, 
Switzerland) 

no 

n/a 

yes 

no 

no 

no 

Scene  Monocular 
(Arrington  Research, 
Scottsdale,  AZ) 

yes 

yes 

yes 

yes 

no 

no 

ASL  Mobile  Eye 
(Applied  Science 
Laboratories:  Bedford, 
MA) 

yes 

no 

yes 

yes 

no 

no 

Deja  View 
(openEyes:  Ames, 

IA) 

yes 

yes 

yes 

no 

yes 

yes 

Eye  Writer  (EyeWriter 
Initative:  New  York) 

no 

yes 

yes 

no 

yes 

yes 

Desktop  Eye-trackers 

ASL  Eyetrac  6 
(Applied  Science 
Laboratories:  Bedford, 
MA) 

yes 

no 

yes 

yes 

no 

no 

Easy  Gaze  (Design 
Interactive:  Oviedo, 
FL) 

yes 

yes 

yes 

yes 

no 

maybe 

Tobii  (all)  (Tobii 
Technology:  Sweden) 

yes 

no 

yes 

yes 

no 

no 

Eyelink  1000  (SR 
Research:  Kanata, 
Canada) 

yes 

no 

yes 

yes 

no 

no 

Gaze  Tracker  (ITU 
Gaze  Group: 
Denmark) 

yes 

yes 

yes 

yes 

yes 

yes 

Easy  Gaze  (LC 
Technology, 

Clearwater,  FL) 

yes 

No 

Yes 

No 

No 

no 
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3.5  Recommendations  and  Suggested  Use 


From  the  results  of  the  eye-tracking  requirements  in  the  Table  5  checklist,  the  recommended 
choice  (highlighted  in  grey)  from  a  high  level  perspective  is  some  form  of  desktop,  remote  eye- 
tracker.  Desktop  eye-trackers  are  the  least  invasive,  as  there  is  nothing  touching  the  user,  and 
require  no  external  setup  aside  from  calibration.  The  goggle  style  eye-tracker  is  less  invasive  than 
the  headmounted,  eye-tracker  although  it  still  does  partially  obstruct  the  user’s  view. 

The  recommended  eye-trackers  based  on  the  selected  criteria  for  use  with  the  IEDD  course  would 
be  Design  Interactive’s  EasyGaze,  or  the  Gaze  tracker.  Both  of  these  brands  are  desktop 
models,  they  are  affordable,  commercially  available,  offer  support,  have  been  used  in  research 
and  can  be  left  at  the  military  base.  The  Deja  View  is  recommended  as  a  third  choice  of  eye- 
tracker  for  use  in  the  course,  as  it  appears  to  be  purchasable,  buildable,  and  minimally  obtrusive. 
Nevertheless,  it  is  possible  that  the  requirements  defined  here  will  change  depending  on  the 
upcoming  budget  and  course  requirements.  If  so,  the  following  list  outlines  some  general 
guidelines  that  should  be  considered  when  purchasing  an  eye-tracker: 

•  If  a  premium,  research- supported  product  is  desired,  and  the  budget  for  eye-tracking 
technology  is  large,  it  is  recommended  to  purchase  a  product  from  Tobii,  Applied  Science 
Laboratories,  or  SR  Research.  These  brands  offer  premium  products  that  are  used 
extensively  in  peer-reviewed  academic  research,  and  offer  extended  support  and  training  for 
their  products.  Also,  it  is  likely  that  these  eye-trackers  could  be  re-used  in  the  future  for 
related  projects. 

o  Alternatively,  if  DRDC  Toronto  is  able  to  borrow  the  Eyelink  II  from  SR  Research, 
it  is  recommended  that  this  product  be  used. 

o  A  second  alternative  is  to  consider  renting  an  eye-tracker  for  the  duration  of  the 
project.  This  could  be  a  more  affordable  option  to  allow  use  of  a  premium  eye- 
tracker  to  calibrate  the  low-cost  alternatives  that  will  remain  at  the  course. 

•  If  the  budget  for  eye-tracking  technology  is  low,  it  is  recommended  that  DRDC  Toronto 
purchase  a  product  from  Easy  Gaze,  or  build  its  own  tracker  using  the  open-source  analysis 
software  EasyGaze.  This  software  is  made  by  Design  Interactive,  and  has  already  been 
purchased  by  the  client,  making  it  an  obvious  choice. 

o  As  one  main  eye-tracker  has  already  been  purchased,  it  is  suggested  that  DRDC 
Toronto  purchase  and  build  low  cost  eye-trackers  to  leave  for  the  IEDD  course,  as  it 
is  unlikely  that  the  EasyGaze  would  remain  in  CFB  Gagetown.  In  this  case,  the 
primary  eye-tracker  could  be  used  to  gather  baseline  data  and  calibrate  the  lower 
quality  eye-trackers. 

•  If  the  ITS  is  to  be  on  a  PC  or  in  a  laptop-style  environment,  it  is  recommended  that  a  desktop 
(e.g.,  EyeGazer)  or  webcam  style  eye-tracker  be  used,  as  headmounted  eye-trackers  are 
invasive  and  more  difficult  to  set  up.  The  trade-off  when  using  a  desktop  or  webcam  style 
eye-tracker  may  have  a  less  accurate  pixel  capture  and  re-calibration  capability  if  the  user  is 
distracted.  However,  the  intrusive,  unwieldy  headgear  associated  with  headmounted  eye- 
trackers  may  deter  their  use  altogether. 
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•  If  the  ITS  will  be  used  in  a  real  life,  or  virtual  immersive  environment  where  the  user  is 
physically  moving  within  that  environment,  a  goggle  style  eye-tracker  (e.g.,  Mobile  Eye)  is 
recommended,  as  these  types  of  eye-trackers  are  built  for  indoor/outdoor  lighting  conditions, 
and  allow  the  participant  to  move  freely  about  in  the  environment. 

Of  note  for  upcoming  technological  implementations  is  that  the  chosen  ITS  eye-tracker  will  need 
to  provide  real-time  x-  and  y-  coordinate  output  of  eye-position  to  determine  what  the  user  is 
actually  looking  at.  Typically,  in  eye-tracking  research,  the  eye-tracker  produces  heat  maps  of 
gaze  patterns  which  are  analyzed  retrospectively  by  the  researcher.  However,  for  the  ITS,  the 
researcher’ s  static  knowledge  of  participant  gaze  is  not  sufficient.  In  this  case,  it  is  proposed  that 
the  data  be  analyzed  in  real  time,  which  could  potentially  pose  integration  problems,  and  increase 
the  overall  complexity  significantly. 

To  attempt  to  remedy  this  issue,  some  type  of  software  developer’s  kit  (SDK)  will  need  to  be 
acquired  with  whatever  eye-tracker  is  used  in  order  to  feed  the  resulting  x-  and  y-  data  into  the 
corresponding  ITS.  The  accessibility  of  the  analysis  software  (i.e.,  open  source  of  the  architecture 
code)  will  determine  whether  the  ITS  will  then  have  to  interpret  the  meaning  of  the  eye  position, 
or  if  a  pre-packaged  analysis  software  could  provide  these  data. 

From  a  technical  standpoint,  it  is  possible  that  eye-tracking  at  this  early  stage,  in  fact  may  be  too 
complex  and  time  consuming  to  feasibly  incorporate  into  a  proof  of  concept  project.  Ultimately,  it 
is  strongly  recommended  that  the  actual  ability  and  effort  required  to  merge  real-time  eye¬ 
tracking  data  into  an  adaptive  system  be  considered. 
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4  Cognitive  Learning  Styles 


As  the  CF  demonstrate  their  readiness  for  distance  and  e-based  learning,  it  is  imperative  to 
understand  the  challenges  that  may  accompany  the  technology  and  approaches  used.  According  to 
DRDC  Toronto,  one  of  the  most  difficult  activities  for  a  distance  education  facilitator  will  be  to 
provide  the  same  or  higher  degree  of  responsiveness  to  the  student  as  would  a  classroom 
facilitator,  and  to  customize  the  learning  experience  to  each  student’s  individual  learning  style. 
One  of  the  mechanisms  to  facilitate  the  learning  experience  is  intelligent  tutoring  technology.  To 
make  the  technology  effective  in  a  distance  learning  environment,  student  learning  styles  must  be 
investigated  prior  to  implementing  any  technology  in  order  to  facilitate  the  customization  of  each 
student’s  learning  experience. 

This  section  reviews  relevant  literature  and  technology  related  to  learning  styles  in  order  to  make 
recommendations  and  suggested  use  for  implementation  within  an  ITS  for  the  IEDD  course. 

4.1  Introduction  to  Learning  Styles 

Research  has  shown  that  different  students  approach  learning  tasks  and  interact  with  learning 
environments  in  different  ways.  As  such,  each  student  develops  a  specific  set  of  learning 
behaviours  to  which  he  or  she  becomes  accustomed.  Such  viewpoints  have  led  to  suggestions  of 
tailoring  educational  interactions  to  students’  cognitive  or  learning  styles  in  the  context  of 
computer  and  web-based  learning  environments  (for  a  review,  see  Pronovost,  Roberts  &  Banbury, 
2008).  The  flexibility  offered  by  such  environments  should  enhance  learning,  allowing  students  to 
develop  personal  navigation  patterns  and  interaction  behaviours  that  reflect  on  their  own 
cognitive  characteristics. 

Cognitive  and  learning  styles  refer  to  roughly  overlapping  yet  distinct  theoretical  constructs 
employed  by  a  number  of  diverse  research  fields  related  to  the  topic  of  learning;  such  as  cognitive 
psychology,  educational  psychology,  personality  psychology,  psychoanalysis,  neuropsychology, 
and  cognitive  and  behavioural  neuroscience. 

Cognitive  style  can  be  defined  as  “an  individual’s  characteristic  and  consistent  approach  to 
organizing  and  processing  information”  (Tennant,  1988).  From  this  perspective,  cognitive  style  is 
considered  to  be  a  central  and  unchanging  part  of  the  individual’s  personal  and  psychological 
makeup  or  “a  fixed  characteristic  of  an  individual”  (Riding,  1996).  For  example,  Anderson  (2004) 
defines  cognitive  learning  styles  as: 

"...  the  information  processing  habits  of  an  individual.  Unlike  individual  differences  in 
abilities,  cognition  describes  a  person's  typical  mode  of  thinking,  perceiving, 
remembering,  or  problem  solving.  Cognitive  style  is  usually  described  as  a  personality 
dimension  which  influences  attitudes,  values,  and  social  interaction.  For  example,  ask 
yourself  how  you  process  experiences  and  knowledge  and  how  you  organize  and  retain 
information.  Do  you  need  to  visualize  the  task  before  starting?  Do  you  approach  learning 
and  teaching  sequentially  or  randomly?  Do  you  work  quickly  or  deliberately?  These  are 
examples  of  cognitive  learning  style  characteristics.  The  biological  basis  for  cognitive 
learning  styles  is  grounded  in  brain  theory.”  (Anderson,  2004). 
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It  should  be  noted  that  the  terms  cognitive  style  and  cognitive  learning  style  are  often  used 
interchangeably  in  the  literature,  and  refer  to  the  same  construct.  Importantly,  cognitive  style  (or 
cognitive  learning  style)  differs  from  learning  style  as  the  latter  can  be  considered  to  vary  over 
time  and  space  (Valley,  1997),  It  is  important  to  note  that  in  this  report,  we  use  the  term 
“Learning  Styles”  to  refer  to  all  cognitive  and  learning  styles. 

4.2  Synthesis  of  Cognitive  Learning  Styles  Knowledge 

Recently,  DRDC  Toronto  started  a  survey  and  synthesized  the  body  of  knowledge  on  student 
learning  styles  and  intelligent  tutoring  technology.  This  review  was  then  used  to  develop 
recommendations  on  how  to  integrate  knowledge  of  the  user’s  learning  style  into  adaptive 
learning  and  intelligent  tutoring  technologies  so  as  to  facilitate  a  positive  learning  experience 
(Hou,  Sobieraj,  Pronovost,  Roberts,  &  Banbury,  2010).  In  this  investigation,  13  of  the  most 
influential  cognitive  and  learning  styles  were  critiqued  as  based  on  the  comprehensive  in-depth 
review  by  Coffield,  Moseley,  Hall,  and  Ecclestone  (2004a,  2004b)  of  7 1  cognitive  and  learning 
styles.  To  date,  the  review  of  Coffield  et  al.  is  the  single  and  most  impartial  review  of  the 
literature  on  cognition,  learning,  and  pedagogy. 

The  learning  styles  reviewed  in  the  DRDC  report  were  based  on  the  criteria  of  being  widely 
quoted  and  central  to  the  field  as  a  whole,  having  a  basis  in  explicit  theory,  having  publications 
that  were  representative  of  the  literature  and  the  total  range  of  models  available,  the  theory  having 
been  proven  to  be  productive  (i.e.,  leading  to  further  research  by  others),  and  the 
instruments/questionnaires/inventory  having  been  widely  used  by  practitioners,  teachers,  tutors  or 
managers.  Table  6  is  an  example  of  some  of  the  13  key  learning  styles  that  were  examined  in  the 
report  by  Hou  et  al.,  (2010),  in  addition  to  their  respective  strengths  and  weaknesses  as  based  on 
the  reviews  by  Coffield  et  al. 


16 


DRDC  Toronto  TR  2010-074 


Table  6.  Excerpt  of  reviewed  learning  styles  from  Hou  et  al.  (2010) 


Learning  Style 

Advantages 

Weaknesses 

Allinson  and  Hayes’ 
Cognitive  Styles 

Index  (CSI) 

•  Best  evidence  for  reliability  and 
validity. 

•  The  constructs  of  analysis  and 
intuition  are  relevant  to  decision 
making  and  work  performance 
in  many  contexts,  although  the 
pedagogical  implications  of  the 
model  have  not  been  fully 
explored. 

•  The  CSI  is  a  suitable  tool  for 
researching  and  reflecting  on 
teaching  and  learning,  especially 
if  treated  as  a  measure  of  two 
factors  rather  than  one. 

•  Matched  styles  are  often 
effective  in  mentoring 
relationships 

•  The  proposed  single  dimension  is  very 
broad  and  made  up  of  diverse,  loosely 
associated  characteristics. 

•  Despite  the  claims  of  its  authors,  the 
CSI  has  been  shown  to  measure  two 
related,  albeit  multi-faceted, 
constructs. 

•  The  popularized  stereotype  of  left-  and 
right-brained-ness  creates  an  unhelpful 
image  of  people  going  through  life 
with  half  of  their  brains  inactive.  . 

•  Intuition  and  analysis  are  not  opposite, 
mutually  exclusive  features 

Entwistle’s 
Approaches  of  Study 
Skills  Inventory  for 
Students  (ASSIST) 

•  Model  aims  to  encompass 
approaches  to  learning,  study 
strategies,  intellectual 
development  skills  and  attitudes 
in  higher  education. 

•  Considerable  literature 
validating  the  model  and 
theoretical  background. 

•  Teachers  and  learners  can  share 
ideas  about  effective  and 
ineffective  strategies  for  learning 

•  Complexity  of  the  developing  model 
and  instruments  is  not  easy  for  non¬ 
specialists  to  access. 

•  Danger  of  categorizing  or  stereotyping 
learners’  characteristics  if  theory  and 
model  not  known  in  depth. 

•  There  is  a  large  gap  between  using  the 
instrument  and  transforming  the 
pedagogic  environment 

Kolb’s  Learning 

Style  Inventory  (LSI) 

•  Lairly  detailed  history  of 
revisions  and  reviews. 

•  Theory  based  on  explicit 
assumptions 

•  Contradictory  and  inconclusive 
findings. 

•  Issues  on  reliability  and  validity. 

•  Concept  of  learning  cycles 
controversial 

Myers-Briggs’  Type 
Indicator  (MBTI) 

•  Lace  validity  is  uncontroversial, 

limited  evidence  of  positive 
pedagogical  implications  of 
matching  learning  style  between 
learners  and  educators 

•  Unclear  implications  for  pedagogy, 
not  a  performance  predictor. 

•  Construct  validity  is  contested 

The  DRDC  report  (Hou  et  al.,  2010)  concluded  that  there  was  a  lack  of  substantial  evidence  to 
select  any  one  theory,  arguing  that  the  review  of  Coffield  et  al  of  cognitive  and  learning  styles 
was  not  definitive,  nor  was  it  conclusive  about  most  of  the  theories,  models,  and  psychometric 
tools  reviewed  as  a  function  of  the  lack  of  independent  validation  data.  On  the  positive  side, 
Coffield  et  al.  acknowledge  that  pragmatically-oriented  concerns,  such  as  Kolb’s,  Entwistle’s,  and 
Vermunt’s  interest  in  changing  the  whole  teaching  -  learning  environment,  beyond  considerations 
to  individual  differences  in  learning  styles,  should  be  pursued  for  the  betterment  of  education  and 
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pedagogy.  Conversely,  other  opportunities  for  the  implementation  of  cognitive  and  learning  style 
models  (e.g.,  career  counselling  and  personnel  selection)  are  not  recommended  as  based  on  the 
observation  that  the  psychometric  tools  available  remain  largely  unvalidated.  Furthermore,  the 
concepts  of  style  matching,  or  deliberate  mismatching  of  learning  styles  between  students 
working  in  groups  or  between  students  and  tutors,  are  consistently  unsupported  by  research. 
While  it  is  intuitively  appealing,  no  evidence  suggests  an  increase  in  performance. 

The  most  vehement  criticism  by  Coffield  et  al  relates  to  the  assertion  that  learning  styles  ought  to 
be  significant  to  a  certain  degree  that  matters  for  education  and  pedagogy  (i.e.,  validation  data 
show  meaningful  effect  sizes).  However,  very  few  reviewers  have  actually  measured  effect  sizes, 
whether  by  using  Pearson’s  R  correlation  (when  the  data  are  continuous  or  binary)  and  its 
accompanying  coefficient  of  determination  ( R 2,  a  measure  of  the  proportion  of  variance  shared  by 
the  two  variables),  d  (in  the  context  of  a  t-test  on  means)  (Cohen,  1988)  or  eta  squared  (r|2,  the 
proportion  of  variance  explained  in  an  analysis  of  variance).  Those  studies  that  have  done  so 
show  disappointing  results. 

In  summary,  the  criticisms  of  learning  styles  complied  by  Coffield  et  al.  relate  to: 

•  The  presence  of  some  theoretical  incoherencies  and  conceptual  confusions  in  the  constructs 
and  factorial  designs  of  such  constructs; 

•  Practical  issues  related  to  learning  styles  such  as  labeling  and  stereotyping,  as  well  as  some 
vested  interests  from  the  authors; 

•  The  variable  quality  of  learning  style  models; 

•  Widespread  psychometric  weaknesses  derived  from  the  learning  style  models; 

•  The  unwarranted  faith  placed  in  simple  inventories; 

•  No  clear  implications  for  pedagogy;  and, 

•  The  lack  of  communication  between  different  research  perspectives  on  pedagogy. 


In  response  to  the  inconclusive  findings  of  Coffield  et  al.,  this  review  was  extended  in  an  attempt 
to  locate  new  learning  styles  that  were  not  reviewed  by  them,  or  to  provide  more  concrete 
evidence  for  the  reliability  and  validity  of  the  styles  that  had  already  been  covered.  The  search 
yielded  the  Index  of  Learning  Styles  (ILS;  Felder  &  Silverman,  1988;  Felder  &  Solomon,  2006). 
The  ILS  was  also  briefly  touched  upon  for  use  in  intelligent  tutoring  systems  by  Banbury  et  al. 
(2009);  however,  the  single  reference  provided  was  insufficient  to  justify  the  ILS’s  sole  use  in  the 
current  recommendations.  As  a  result,  this  review  has  included  updated  and  detailed  support  for 
the  use  of  the  ILS  within  the  proposed  ITS  for  the  IEDD  course. 

4.3  Detailed  Review  of  Index  of  Learning  Styles  (ILS) 

This  sub-section  provides  an  overview  of  the  Felder-Solomon  Index  of  Learning  Styles.  In 
addition,  this  section  introduces  an  adaptive  learning  aid  (LOCATE™),  which  is  a  software  that 
was  developed  within  DRDC  to  aid  in  the  design  of  workspaces  using  learning  styles.  Although 
this  learning  aid  does  not  specifically  use  the  ILS,  the  learning  style  dimensions  assessed  by 
LOCATE™  do  appear  to  be  very  similar  to  those  assessed  by  the  ILS.  As  such,  LOCATE™  was 
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deemed  important  to  include  in  this  section  as  it  is  an  internally  designed  application  which  may 
be  of  interest  to  distance  education  and  e-learning  in  the  CF. 

4.3.1  Overview  and  design  of  model 

Felder  and  Solomon’s  Index  of  Learning  Styles  (Felder  &  Solomon,  2006),  is  an  instrument  used 
to  assess  preferences  on  four  dimensions  of  a  learning  style  model  that  was  formulated  by  Felder 
and  Silverman  (1988).  The  ILS  can  be  classified  as  resting  within  the  category  of  learning  styles 
having  flexible  traits  (Coffield  et  al.,  2004a;  2004b).  It  was  developed  based  on  the  belief  that  the 
primary  goal  of  a  learning  style  model  should  be  to  provide  guidance  to  instructors  on  how  to 
develop  a  balanced  teaching  method  that  addresses  the  needs  of  students  with  diverse  learning 
style  preferences  (Felder,  Litzinger,  Lee,  &  Wise,  2005). 

The  ILS  consists  of  44  items,  broken  down  into  four  scales  of  1 1  questions,  with  each  scale 
corresponding  to  one  of  the  four  dimensions  of  the  learning  style  model  (Felder  &  Silverman, 
1988).  To  note,  each  of  the  four  dimensions  contains  a  set  of  two  opposite  categories.  The  idea 
behind  these  opposite  categories  is  that  everyone  uses  all  of  them  at  different  times,  but  with 
varying  degrees  of  preference. 

The  four  dimensions  in  the  ILS  are: 

•  The  Active/Reflective  Dimension:  How  do  you  prefer  to  process  information? 

o  Active  learners  prefer  to  process  information  by  talking  about  it  and  trying  it  out 
(e.g.,  they  prefer  active  student  participation  in  groups). 

o  Reflective  learners  prefer  to  think  about  information  before  acting  (e.g.,  they  prefer 
passive  student  participation  by  themselves  or  with  one  familiar  partner). 

•  The  Sensing/Intuitive  Dimension:  How  do  you  prefer  to  take  in  information? 

o  Sensing  learners  prefer  to  take  in  information  that  is  concrete  and  practical. 

o  Intuitive  learners  prefer  to  take  in  information  that  is  abstract,  and  more  conceptual  in 
nature. 

•  The  Visual/Verbal  Dimension:  How  do  you  prefer  information  to  be  presented? 

o  Visual  learners  prefer  visual  presentations  of  material:  diagrams,  charts,  graphs, 
pictures. 

o  Verbal  learners  prefer  explanations  with  words,  in  the  form  of  both  written  and 
spoken  presentations. 

•  The  Sequential/Global  Dimension:  How  do  you  prefer  to  organize  information? 

o  Sequential  learners  prefer  to  organize  information  in  a  linear,  orderly  and  systematic 
fashion. 
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o  Global  learners  prefer  to  organize  information  more  holistically  and  in  a  seemingly 
scattered  and  disorganised  manner. 

As  previously  mentioned,  each  dimension  consists  of  two  categories  of  opposite  preferences,  and 
each  category  has  a  score  ranging  from  1  to  11.  In  the  ILS,  students  complete  a  sentence  by 
selecting  one  of  two  response  options  representing  opposite  ends  of  one  of  the  learning  styles 
scales.  Scores  ranging  from  1  to  3  indicate  that  the  student  is  well  balanced  between  the  two 
categories  of  a  particular  learning  styles  dimension.  For  scores  between  5  and  7,  a  moderate 
preference  is  indicated,  which  means  favouritism  for  one  of  the  two  categories.  Scores  between  9 
and  11  indicate  a  very  strong  preference,  meaning  that  the  student  will  have  difficulty  with 
learning  in  an  environment  that  does  not  support  that  preference. 

4.3.2  Reliability  and  validity 

Felder  and  Spurlin  (2005)  conducted  the  first  comprehensive  examination  of  the  ILS  and  assessed 
the  reliability  and  validity  of  21  external  studies  using  the  instrument.  These  analyses  were 
conducted  by  examining  Pearson  correlation  coefficients.  To  note,  the  correlation  coefficient  is  a 
statistic  that  represents  how  closely  two  variables  co-vary,  or  the  extent  to  which  changes  in  one 
variable  are  associated  with  changes  in  the  other  variable.  Importantly,  the  correlation  coefficient 
indicates  the  strength  and  direction  of  the  relationship  between  two  variables.  It  can  vary  from  -1 
(perfect  negative  correlation)  through  0  (no  correlation)  to  +1  (perfect  positive  correlation). 

The  Felder  and  Spurlin  review  indicated  that  the  test  -  rest  reliability  of  the  ILS  was  acceptable  as 
it  fell  in  the  correlation  ranges  of  .73  to  .87  after  4  weeks,  and  .56  to  .77  after  10  weeks.  Ah 
correlation  coefficients  were  significant  at  the  0.05  level  or  better  thus  indicating  that  participant 
responses  to  the  ILS  did  not  change  greatly  over  time.  The  internal  consistency  of  the  four 
dimensions  ranged  from  .51  to  .62  for  Active/Reflective,  from  .65  to  .76  for  Sensing/Intuitive, 
from  .56  to  .69  for  Visual/Verbal,  and  from  .41  to  .54  for  Sequential/Global.  These  reliability 
coefficients  all  meet  the  minimal  standard  of  .50  that  was  suggested  by  Tuckman  (1999)  for 
preference  and  attitude  assessments.  They  also  suggest  that  the  test  items  for  each  subscale  were 
effectively  tapping  into  their  target  dimension. 

A  factor  analysis  was  conducted  with  the  ILS  and  revealed  that  the  Active/Reflective, 
Sensing/Intuitive,  and  Visual/Verbal  dimensions  are  orthogonal.  The  Sequential/Global  and 
Sensing/Intuitive  dimensions  were  found  to  be  associated,  and,  thus,  assessing  both 
dimensions  may  lead  to  redundancy.  Pearson  correlation  coefficients  relating  preferences  on  the 
different  dimensions  of  the  ILS  in  four  studies  were  consistently  .2  or  less  except  for 
Sensing/Intuitive  and  Sequential/Global  dimensions  (which  ranged  from  .32  to  .48).  Again,  this 
suggests  that  the  Sensing/Intuitive  and  Sequential/Global  dimensions  may  be  assessing  many  of 
the  same  preferences. 

4.3.3  Implications  for  pedagogy  and  evidence  of  pedagogical  impact 

Unlike  many  of  the  other  learning  styles  theorists,  Felder  and  Solomon  are  not  proponents  of 
directly  matching  educational  strategies  and  pedagogical  tools  to  individual  learning  styles. 
Instead,  they  assert  that  most  students  learn  differently  than  their  instructors,  and  other  students. 
Thus,  it  becomes  impossible  for  an  instructor  to  simultaneously  address  the  learning  needs  of  all 
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students.  Additionally,  an  instructor’s  preferred  method  of  teaching  may  be  influenced  by  his/her 
own  learning  style  preferences.  Instructors  must  be  mindful  of  these  personal  biases  in  addition  to 
being  aware  of  the  diverse  learning  needs  of  their  students.  The  most  effective  instructors  will  be 
those  who  can  present  material  using  the  widest  array  of  teaching  methods,  thereby  catering  to  as 
many  learning  style  preferences  as  possible. 

Several  researchers  have  cited  positive  student  outcomes  following  the  introduction  of 
multifaceted  pedagogical  tools  and  methods  that  cater  to  students  with  diverse  learning  styles. 
Felder  (1995)  investigated  the  performance  of  chemical  engineering  students  who  were  exposed 
to  novel  instructional  methods  (e.g.,  use  of  realistic  examples,  field  experiences,  guest  speakers, 
etc.)  or  more  traditional  instructional  methods  (e.g.,  long  lectures,  homework  assignments,  etc.). 
It  is  important  to  note  that  the  novel  instructional  methods  addressed  a  wider  array  of  learning 
style  needs  than  did  traditional  instructional  methods,  which  cater  to  students  with  verbal  and 
sequential  learning  style  needs  (i.e.,  long  verbal  lectures  and  homework  assignments  requiring 
step  by  step  problem  solving).  Felder  documented  that  students  in  the  novel  instruction  group 
exhibited  superior  performance  as  compared  to  students  receiving  traditional  instruction.  For 
example,  these  students  showed  greater  proficiency  in  generating  creative  solutions  to  problems, 
enhanced  teamwork  skills,  and  an  increased  likelihood  of  attending  graduate  school. 

In  a  related  study,  Tripp  and  Moore  (2007)  introduced  pre-service  elementary  school  teachers  to 
the  ILS  and  instructed  them  on  how  knowledge  of  learning  styles  can  be  incorporated  into 
teaching  strategies.  The  pre-service  teachers  reported  that  after  gaining  greater  awareness  of 
learning  styles,  they  felt  more  sensitive  to  the  needs  of  their  students,  and,  in  turn,  this  enabled 
them  to  prepare  better  lesson  plans.  This  suggests  that  instructors  should  also  be  assessed  with  the 
ILS,  so  that  they  can  compare  their  own  styles  with  those  of  their  students,  which  in  turn  will  lead 
to  positive  pedagogical  outcomes. 

4.3.4  Applications 

The  ILS  is  intended  for  use  with  adults,  especially  higher  education  students  and  their  instructors. 
It  is  important  to  remember  that  proponents  of  the  ILS  do  not  advocate  the  necessity  of  matching 
student  and  instructor  learning  styles,  as  doing  so  would  be  quite  arduous  due  to  the  high  number 
of  possible  learning  style  combinations.  Instead,  proponents  advocate  the  use  of  teaching  methods 
that  present  material  in  a  wide  variety  of  ways,  thereby  catering  to  diverse  learning  style 
preferences.  In  turn  this  leads  to  positive  performance  outcomes  for  students  and  heightened 
feelings  of  effectiveness  for  instructors  (Felder,  1995;  Tripp  &  Moore,  2007). 

There  are  no  apparent  suggestions  for  use  of  the  ILS  in  the  workplace.  However,  several  e- 
learning  applications  have  been  put  forth.  Kim,  Kim,  Cho,  and  Park  (2005)  designed  an 
intelligent  learning  environment  where  the  individual’s  learning  style  preferences  were  diagnosed 
through  their  activity  patterns  on  a  webpage.  Subsequently,  individual  user  interfaces  were 
customized  in  an  adaptive  manner  to  accommodate  these  preferences.  In  this  way,  the  e-learning 
program  was  able  to  present  learning  content  in  a  way  that  appealed  to  all  learning  styles. 
Similarly,  Graf  and  Kinshuk  (2007)  went  one  step  further  by  assessing  the  effectiveness  of 
adaptive  e-learning  systems.  Students  were  randomly  assigned  to  one  of  three  groups: 


DRDC  Toronto  TR  2010-074 


21 


■  The  “matched”  group  was  presented  with  a  course  that  matched  their  learning  style; 

■  The  “mismatched”  group  was  presented  with  a  course  that  mismatched  their  learning 
style;  and 

■  The  “standard”  group  was  presented  with  a  course  in  a  sequence  that  was  independent  of 
their  learning  style.  The  researchers  found  that  students  from  the  “matched”  group  spent 
less  time  in  the  course  but  achieved  on  average  the  same  scores  as  students  in 
the  other  groups. 

The  authors  suggested  that  this  quicker  completion  time  among  students  in  the  “matched”  group 
is  an  indication  of  their  heightened  satisfaction  with  the  course  as  due  to  an  increased  ease  of 
interaction  with  the  e-learning  system. 

4.3.5  Application  of  the  ILS  at  DRDC:  LOCATE™ 

LOCATE™  (Edwards,  2005;  Scott  &  Edwards,  2006;  Edwards  &  Scott,  2007)  represents  an 
application  of  the  ILS  within  the  specific  context  of  DRDC’s  adaptive  learning  requirements.  It  is 
a  software  tool  that  aids  in  the  design  of  workspaces  using  learning  styles.  Even  though  it  does 
not  specifically  use  the  ILS,  the  learning  style  dimensions  that  it  does  assess  are  similar  to  those 
assessed  by  the  ILS. 

LOCATE™  supports  the  design,  analysis,  and  optimization  of  workspace  layouts  based  on  the 
type  and  nature  of  the  work  to  be  conducted.  One  key  aspect  of  LOCATE™  is  an  adaptive  help 
system,  which  plays  a  key  role  in  DRDC  Toronto’s  ongoing  efforts  to  develop  and  refine  adaptive 
learning  technologies.  LOCATE™’ s  user  model  contains  information  about  the  user’s 
knowledge,  preferences,  abilities,  and  learning  style,  which  enables  the  software  to  make 
informed  decisions  about  the  style  of  help  that  it  offers  to  its  user. 

To  assess  learning  style,  LOCATE™  asks  users  to  answer  a  set  of  questions,  as  based  on  the 
Cognitive  Style  Questionnaire  (CSQ)  that  was  developed  by  Edwards  (2005).  According  to  the 
CSQ,  learning  styles  fall  along  a  Verbal,  Imagery,  and  Kinaesthetic  tri-mension  and  a  Holistic 
and  Analytic  dimension  (see  Figure  1). 
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Figure  1.  Cognitive  style  modelled  as  a  triangular  solid 


It  is  important  to  note  that  the  learning  style  preferences  assessed  by  the  CSQ  are  very  similar  to 

those  assessed  by  the  ILS: 

•  Verbal/Analytic:  Textual  descriptions  of  how  tasks  are  performed;  Vcrbal/Wholist:  Textual 
descriptions  of  task  performance,  including  contextual  information.  (Similar  to  ILS’s 
Visual/Verbal  dimension); 

•  Imagery/ Analytic:  Animated  demonstrations  in  which  the  software  shows  exactly  how 
a  task  is  performed,  directly  in  LOCATE™’ s  interface;  Imagery /Wholist:  Graphical 
instructions,  where  the  steps  in  carrying  out  a  task  are  illustrated  with  a  sequence  of  still 
images  and  contextual  information  on  the  feature  is  available.  (Similar  to  ILS’s 
Global/Sequential  dimension); 

•  Kinaesthetic/ Analytic:  Practice  sessions  in  which  users  can  try  out  a  feature  in  LOCATE™ 
as  they  learn  about  it;  Kinaesthetic/Wholist:  Practices  session  in  which  users  can  try  out 
LOCATE™’ s  features,  with  additional  information  on  the  context  in  which  those  features 
are  used.  (Similar  to  ILS’s  Sensing/Intuitive/  and  Active/Reflective  dimensions). 
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The  information  that  is  derived  from  user  responses  to  the  CSQ  is  stored  in  the  user 
model.  As  help  is  requested  by  the  user,  the  help  material  is  provided  in  a  format  that  is  most 
supportive  of  the  user’s  learning  style.  Importantly,  LOCATE™  continuously  adapts  the  format 
of  the  help  system  to  the  user’s  learning  preferences  by  tracking  the  user’s  behaviour  as  he  or  she 
selects  alternatives. 

Importantly,  the  current  work  of  Edwards  (2005),  and  Edwards  and  Scott  (2007)  involving 
LOCATE™  and  learning  space  in  the  design  of  workspace  layouts  and  user  “help”  functions  is 
still  in  need  of  validation  and  further  testing.  One  recommendation  that  can  be  taken  from  the 
work  on  LOCATE™  is  the  integration  of  learning  styles  into  the  user’s  help  menu  when 
designing  adaptive  distance  education  and  e-learning  technologies  for  use  with  the  CF.  In  this 
way,  the  learning  styles  of  the  user  would  be  assessed  at  the  start  of  the  learning  session,  and 
whenever  the  user  would  subsequently  request  help  from  the  system,  options  would  be 
provided  in  the  user’s  preferred  style  of  help  (e.g.,  visual,  text,  video  clips,  interactive, 
etc.).  In  addition,  options  could  be  provided  to  the  user  to  select  help  in  a  form  other  than  the 
system’s  recommended  style,  which  would  allow  for  the  system  to  adapt  to  the  user’s  preferred 
style  of  help. 

4.3.6  Overall  assessment 

A  review  of  the  literature  suggests  that  the  immediate  advantage  of  the  ILS  is  that  it  encompasses 
the  advantages  of  several  of  the  previously  reviewed  learning  styles,  and  also  has  recent  data  to 
support  its  reliability  and  validity  (Felder  &  Spurlin,  2005;  Palapu,  2007).  However,  a  factor 
analysis  did  reveal  low  orthogonality  between  the  Sequential/Global  and  Sensing/Intuitive 
dimensions  of  the  ILS,  thereby  suggesting  that  there  may  be  some  redundancy  between  these  two 
dimensions.  In  terms  of  pedagogy,  researchers  have  documented  that  the  use  of  the  ILS  does  lead 
to  positive  pedagogical  outcomes,  with  heightened  performance  among  students  and  enhanced 
self-reported  sensitivity  to  student  needs  among  instructors  (e.g.,  Tripp  &  Moore,  2007). 
Importantly,  the  ILS  has  also  been  incorporated  into  several  e-learning  programs  (e.g.,  Graf  & 
Kinshuk,  2007),  and  has  led  to  positive  student  outcomes.  In  turn,  this  suggests  that  the 
incorporation  of  the  ILS  into  computer-based  learning  and  distance  education  may  lead  to 
favourable  learning  outcomes.  It  remains  to  be  seen  whether  or  not  the  ILS  can  be  applied  in 
business  contexts.  Finally,  as  based  on  favourable  outcomes  following  DRDC’s  design  of  a 
learning  aid  called  LOCATE™,  which  is  based  upon  the  ILS,  it  is  recommended  that  the  ILS 
model  should  be  used  to  identify  the  learning  styles  of  CF  learners  undergoing  computer-based 
training  in  a  distance  educational  context  in  order  to  improve  learning  effectiveness. 

4.4  Recommendations  and  Suggested  Use 

•  It  is  recommended  that  the  ILS  (Felder  &  Solomon,  2006)  be  used  as  a  tool  to  identify  the 
learning  styles  of  CF  learners  undergoing  computer-based  training  in  a  distance  educational 
context  in  order  to  customize  their  learning  experience. 

•  A  baseline  of  learning  styles  should  be  assessed  using  the  ILS  online  questionnaire 
administered  to  capture  the  initial  values  used  to  represent  the  learner’s  style. 
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•  Baseline  ILS  styles  should  be  compared  against  the  style  of  current  IEDD  course 
teaching/presentation  styles  to  see  if  the  areas  where  people  are  failing  are  indeed 
those  which  show  a  mismatch  between  learning  and  teaching  style,  and/or 
presentation  of  information. 

•  Given  the  low  orthogonality  between  the  Sequential/Global  and  Sensing/Intuitive 
dimensions  of  the  ILS  (Felder  &  Spurlin,  2005),  and  the  increased  effort  required  to  build 
content  in  multiple  learning  style  formats,  it  is  recommended  that  focus  should  be  given  to 
the  design  of  the  ITS’s  course  content  to  correspond  to  only  three  dimensions  of  the  ILS, 
which  will  reduce  redundancy. 

•  It  is  suggested  as  in  Tripp  and  Moore  (2007),  that  IEDD  course  instructors  also  be  assessed 
with  the  ILS,  and  allowed  to  compare  their  own  styles  with  those  of  their  students.  The  goal 
is  that  teaching  can  be  improved  when  teaching  styles  are  better  matched  with  learning 
styles.  This  suggestion  allows  for  students  of  the  IEDD  course  to  benefit  from  an  integrated 
ILS  within  the  entire  course  content  beyond  the  online  components. 

•  Although  the  current  work  of  Edwards  and  Scott  (2005,  2007)  involving  LOCATE™  and 
learning  styles  in  the  design  of  workspace  layouts  and  user  “help”  functions  is  still  in  need  of 
validation  and  further  testing,  it  is  recommended  to  re-visit  the  underlying  architectures  and 
processes  involved  in  the  building  and  design  of  the  LOCATE™  program. 

o  It  is  currently  beyond  the  scope  of  this  report  to  make  recommendations  about 
computational  architectures  and  other  programming  strategies,  however,  this 
reference  has  been  noted  as  having  potential  value  to  the  current  project,  and  will  be 
considered  in  greater  detail  in  upcoming  projects  relating  to  the  design  of  the  ITS. 

o  Furthermore,  it  should  be  noted  that  the  learning  styles  implemented  in  LOCATE™ 
parallel  those  of  this  report’s  suggested  ILS.  For  instance,  Edwards  (2005) 
distinguishes  among  visual,  textual,  holistic  (global),  active  and  reflective  type 
domains.  As  such,  it  is  recommended  to  check  for  more  recent  references  from 
these  authors  when  the  current  program  is  closer  to  the  actual  design  integration. 

•  One  recommendation  that  can  be  taken  from  the  work  on  LOCATE™,  however,  is  the  use  of 
learning  styles  to  be  integrated  into  the  user’s  help  menu.  The  program  created  a  new 
resource  called  “how  to”  help,  designed  to  provide  procedural  help  in  carrying  out  tasks. 
When  a  user  requests  help  from  the  system,  options  are  provided  to  select  help  in  a  form 
other  than  the  system’s  recommended  style,  which  allows  for  the  system  to  adapt  to  the 
preferred  style  of  help  (e.g.,  visual,  text,  video  clips,  interactive,  etc.) 

o  It  is  recommended  that  a  similar  help  strategy  for  the  IEDD  course  ITS  be  adapted; 
one  suggestion  is  to  incorporate  an  area  where  operators  can  practice  and  “try- 
things  out”,  or  watch  demonstrations  of  interviews  done  correctly  and  incorrectly. 

o  Also,  learning  styles  could  be  incorporated  into  the  hints  given  by  the  tutor. 

o  A  potential  issue  to  note  is  that  the  creation  of  multiple  learning  style  content 

extensively  increases  the  complexity  and  work  expenditure  of  the  programmers 
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involved.  Therefore,  it  is  advisable  to  begin  by  implementing  fewer  learning  styles, 
and  building  onwards  following  a  proof  of  concept. 
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5  Psychophysiological  Measures 


Psychophysiology  deals  with  the  interactions  between  the  mind  and  body  by  recording  how  the 
body  is  functioning  and  relating  these  functions  to  recorded  behaviour.  The  field  is  based  on  the 
premise  that  changes  in  the  body  are  related  to  changes  in  behaviour,  affect  and  motivational 
states  (Blanchard,  Calhoun,  &  Frasson,  2007). 

According  to  Conati  and  Merten  (2007),  psychophysiological  techniques  makes  it  possible  to 
identify  differences  in  human  brain  processing  that  correspond  to  differences  in  learning  styles 
and  capabilities.  These  techniques  also  have  the  potential  to  reveal  the  learners’  cognitive  state. 
Psychophysiological  recording  techniques  are  generally  non-invasive,  that  is,  they  record  from 
the  body's  surface  and  nothing  enters  the  body  of  the  person  that  is  being  recorded. 

Although  there  are  numerous  ways  to  categorize  these  measures  (for  examples,  see  Karamouzis, 
2006),  in  this  section,  psychophysiological  technologies  have  been  categorized  in  terms  of  the 
type  of  measurement  they  provide  (e.g.,  mental  and  physical  workload,  stress,  attention,  etc.). 

There  are  multiple  ways  to  measure  psychophysiological  responses,  and  even  more  technologies 
available  to  do  so.  However,  not  all  available  technologies  and  measurements  are  applicable  for 
the  current  project.  As  such,  the  current  review  focuses  on  the  most  relevant  measures  that  fit  the 
needs  of  the  IEDD  operator  course.  For  instance,  as  the  course  will  be  designed  for  individual  PC 
use,  it  did  not  make  sense  to  include  references  on  measures  of  physical  arousal  such  as 
electromyography.  The  electromyograph  (EMG)  measures  the  electrical  energy  present  during 
muscle  activation. 

This  section  describes  the  relevant  psychophysiological  measures  considered  for  this  report. 
Again,  the  section  ends  with  a  brief  overview  of  the  reference  material,  followed  by 
recommendations  and  suggested  use. 

5.1  Mental  Workload 

A  commonly  researched  variable  of  performance  is  the  assessment  of  mental  workload.  Mental 
workload  is  a  broad  concept  that  refers  to  the  amount  of  processing  capacity  that  is  expended 
during  task  performance  (Eggemeier,  1988).  At  its  core,  mental  workload  refers  to  the  difference 
between  the  processing  resources  available  to  the  operator  and  the  resource  demands  required  by 
the  task  (Sanders  &  McCormick,  1993).  The  term  “workload”  delineates  the  difference  between 
capacities  of  the  human  information  processing  system  that  are  expected  to  satisfy  performance 
expectations,  and  the  capacity  available  for  actual  performance  (Gopher  &  Donchin,  1986). 

Mental  workload  alone  only  describes  a  series  of  measures  of  physiological  indices  produced  by 
the  human  body.  Common  measures  of  mental  workload  are  the  electroencephalogram  (EEG), 
heart  rate  variability  (HRV-  a  measure  involving  the  electrocardiogram  [ECG]),  and  galvanic  skin 
response  (GSR)  either  alone,  or  preferably  in  combination  with  one  another.  These  type  of 
measures  are  useful  metrics  of  mental  workload,  as  they  provide  immediate  feedback  of  mental 
processing  of  the  task  at  hand  (Byrne  &  Parasuraman,  1996;  Gale  &  Christie,  1987;  Kramer, 
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1991;  Parasuraman,  1990).  As  such,  both  type  of  measurements  have  unique  properties  that  make 
them  ideal  for  adaptive  automation. 

Despite  the  many  other  available  metrics  that  have  been  used  to  measure  workload,  this  report 
focuses  on  the  three  aforementioned  metrics  due  to  their  widespread  use  and  accessibility  within 
academic  research,  as  well  as  the  feasible  constraints  on  data  integration  imposed  by  them  in 
consideration  of  their  implementation  into  the  proposed  ITS.  The  following  sub-sections  describe 
EEG,  HRV  (and  ECG)  and  GSR  in  order  to  make  recommendations  and  suggested  uses  for  the 
IEDD  course  ITS. 

5.2  Electroencephalogram  (EEG) 

The  EEG  records  the  electrical  activity  along  the  scalp  produced  by  the  firing  of  neurons  within 
the  brain  (Niedermeyer  &  Silva,  1999).  The  EEG  is  a  method  of  capturing  and  measuring  brain 
waves  to  indicate  how  the  brain  functions  over  time.  In  the  conventional  scalp  EEG,  the  recording 
is  obtained  by  placing  electrodes  on  the  scalp  with  a  conductive  gel  or  paste,  usually  after 
preparing  the  scalp  area  by  light  abrasion  to  reduce  impedance  due  to  the  skin  cells  (Abou-Khalil 
&  Musilus,  2006).  Many  systems  typically  use  electrodes,  each  of  which  is  attached  to  an 
individual  wire.  Some  systems  use  caps  or  nets  into  which  electrodes  are  embedded;  this  is 
particularly  common  when  high-density  arrays  of  electrodes  are  needed. 

The  output  from  an  EEG  system  is  recorded  as  a  graph  of  brainwaves  on  a  time  scale,  and  simply 
reveals  rough  brainwave  frequency  and  amplitude.  Measuring  event  related  potentials  (ERPs) 
involves  correlating  the  EEG  brainwave  response  with  a  stimulus  (event)  and  averaging  the  result 
of  dozens  to  thousands  of  stimulus  expositions  together  to  get  a  clear  picture  of  what  electrical 
activity  took  place  upon  presentation  of  that  specific  stimulus  (Karamouzis,  2006). 

5.3  Electrocardiogram  (ECG) 

The  ECG  records,  non-invasively,  the  electrical  activity  of  the  heart  over  time  typically  by  way  of 
skin  electrodes.  The  heart  emits  the  highest  electrical  activity  of  all  the  body’s  organs,  providing 
robust  physiological  data  about  the  user’s  load  levels  that  might  be  more  difficult  to  detect  via  the 
EEG  alone.  While  the  EEG  provides  information  on  mental  load,  the  ECG  affords  a  robust 
identification  of  motor-related  activity  (Chen  &  Vertegaal,  2004).  According  to  Prinzel,  Freeman, 
Scerbo,  Mikulka,  and  Pope  (2003),  cardiovascular  activity  is  the  most  commonly  used  index  of 
cognitive  workload  as  it  is  relatively  unobtrusive,  reliable,  easy  to  use  and  interpret  (Fahrenberg 
&  Wientjes,  2000),  and  appears  to  be  readily  accepted  by  those  in  an  operational  environment. 

The  ECG  has  been  used  to  assess  a  user’ s  emotional  state  at  a  given  point  in  time,  which  relates 
to  mental  workload  outputs  such  as  anger  or  frustration.  For  example,  McCraty,  Atkinson,  Tiller, 
Rein,  and  Watkins  (1995)  demonstrated  that  specific  emotions  could  be  distinguished  based  upon 
the  power  spectrum  of  the  ECG.  The  other  main  usage  of  the  ECG  is  to  further  calculate  one’s 
HRV.  According  to  McCraty  et  al.,  in  cases  of  stress,  there  is  a  tendency  for  increased  heart-rate 
variability  in  the  lower  frequency  ranges.  McCraty  et  al.  (1995)  claim  that  stress  measures 
provided  by  ECG  low  frequency  components  correlate  well  with  the  mental  load  of  users  during 
complex  visual  tasks  as  measured  by  the  National  Aeronautics  and  Space  Administration  Task 
Load  Index  (NASA  TLX)  -  a  subjective  workload  assessment  tool. 
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5.4  Heart  Rate  Variability  (HRV) 


The  HRV  is  derived  from  the  ECG  and  measures  naturally  occurring,  beat-to-beat  changes  in 
heart  rate  (McCraty,  2003).  A  synthesis  of  publications  under  McCraty  at  the  Institute  of 
HeartMath  Research  Centre  (Boulder  Creek,  CA)  suggests  that  mental  and  emotional  states 
directly  affect  cardiovascular  activity,  and  therefore  the  HRV.  While  the  rhythmic  beating  of  the 
heart  at  rest  was  once  believed  to  be  monotonously  regular,  it  is  now  known  that  the  rhythm  of  a 
healthy  heart  under  resting  conditions  is  actually  irregular.  These  moment-to-moment  variations 
in  heart  rate  are  easily  overlooked  when  average  heart  rate  is  calculated. 

Systems-oriented  models  propose  that  HRV  is  an  important  indicator  of  both  physiological 
resiliency  and  behavioural  flexibility,  reflecting  the  individual’s  capacity  to  adapt  effectively  to 
stress  and  environmental  demands  (McCraty,  2009).  In  a  healthy  individual,  the  heart  rate 
estimated  at  any  given  time  represents  the  net  effect  of  the  parasympathetic  nerves,  which  slow 
heart  rate,  and  the  sympathetic  nerves,  which  accelerate  it  (Tiller,  McCraty,  &  Atkinson,  1996). 
These  changes  are  influenced  by  emotions,  thoughts  and  physical  exercise  (McCraty  et  al.,  1995). 
One’s  changing  heart  rhythms  affect  not  only  the  heart  but  also  the  brain’s  ability  to  process 
information  in  the  processes  of  decision-making,  problem-solving  and  creativity.  Thus,  the  study 
of  HRV  is  a  powerful,  objective  and  non-invasive  tool  used  to  explore  the  dynamic  interactions 
between  physiological,  mental,  emotional  and  behavioural  processes. 

HRV  generally  increases  and  decreases  as  a  function  of  cognitive  workload.  Prinzel  et  al.  (2003) 
reviewed  previous  HR  (heat  rate)/HRV  literature  and  concluded  that  the  research  suggests  that  the 
“mid-band”  HRV  can  accurately  measure  changes  in  mental  workload,  while  retaining  the 
properties  of  diagnosticity,  sensitivity,  reliability,  and  ease  of  use.  Furthermore,  Chen  and 
Vertegaal  (2004)  were  able  to  demonstrate  that  HRV  and  the  EEG  were  directly  related, 
classifying  active  or  passive  participation  in  a  task.  Therefore,  HRV  has  the  potential,  like  the 
EEG  to  be  used  as  a  physiological  “trigger”  for  invoking  adaptive  automation. 

5.5  Galvanic  Skin  Response  (GSR) 

The  GSR,  also  known  as  the  electrodermal  response  (EDR),  psychogalvanic  reflex  (PGR),  or  skin 
conductance  response  (SCR),  can  be  used  as  an  indicator  of  one’s  level  of  excitement  or 
relaxation.  Electro-dermal  activity  measures  have  been  found  to  indicate  mental  effort,  levels  of 
arousal  and  vigilance  and  may  also  respond  to  affective  stimuli  (e.g.,  Chen  &  Vertegaal,  2004). 
When  one  is  mentally,  emotionally,  or  physically  aroused,  a  response  is  triggered  in  the  skin. 
During  excitation,  sweat  glands  in  the  skin  fill  with  salty  sweat  which  acts  as  a  weak  electrolyte 
and  conductor. 

Due  to  the  response  of  the  skin  and  muscle  tissue  to  external  and  internal  stimuli,  the  resistance 
can  vary.  Higher  arousal  (such  as  occurs  with  increased  involvement)  will  almost  instantaneously 
(0.2  -  0.5  seconds)  cause  a  fall  in  skin  resistance;  reduced  arousal  (such  as  occurs  with 
withdrawal)  will  cause  a  rise  in  skin  resistance  (McCleary,  1950). 

When  correctly  calibrated,  the  GSR  can  measure  these  subtle  differences.  There  is  a  relationship 
between  sympathetic  activity  and  emotional  arousal,  although  one  cannot  identify  the  specific 
emotion  being  elicited.  As  such,  it  is  important  to  interpret  any  GSR  output  with  caution,  and  this 
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output  should  be  used  in  multimodal  combination  along  with  other  psychophysiological  measures 
if  one’s  goal  is  to  make  workload  inferences  (Shepherd,  2009). 

As  described  by  Shepherd  (2009),  stimuli  such  as  words  perceived  by  the  human  auditory  system 
carry  emotional  charges  that  influence  one’s  arousal  level,  and  affect  the  skin’s  conductivity. 
Shepherd  goes  on  to  assert  that  the  hands,  representing  a  large  proportion  of  sensitive  nerve 
endings  linking  to  the  sensory-motor  strip  of  the  cortex,  make  hand-held  electrodes  an  ideal  and 
non-invasive  point  of  contact  for  research. 

A  psycho-galvanometer  measures  the  resistance  of  the  skin  to  the  passage  of  a  very  small  electric 
current.  The  two  paths  for  current  are  along  the  surface  of  the  skin  and  through  the  body,  and  the 
device  measures  electrical  resistance  between  two  points  (McCleary,  1950).  The  current  sent 
causes  the  Tarchanoff  Response  -  this  referes  to  a  change  in  direct  current  (DC)  potential  across 
neurons  of  the  autonomic  nervous  system  connected  to  the  sensory-motor  strip  of  the  cortex 
(Shepherd,  2009).  This  change  is  related  to  the  level  of  cortical  arousal  (Ellaway,  Huppuswamy, 
Nicotra  &  Mathias,  2010).  As  arousal  increases,  the  "fight  or  flight"  stress  response  of  the 
autonomic  nervous  system  comes  into  action,  and  adrenaline  causes  increased  sweating  amongst 
many  other  phenomena,.  However,  the  speed  of  the  sweating  response  is  nowhere  near  as 
instantaneous  or  accurate  as  the  Tarchanoff  Response. 

Classic  technologies  used  for  measuring  the  GSR  include  strapping  electrodes  to  the  index  and 
middle  finger,  while  more  recent  research,  such  as  that  of  Handwave  (Strauss,  Reynolds,  & 
Picard,  2005)  and  Brainquiry  ("Neurofeedback  equipment  -  Wireless  Brainquiry  PET  EEG  and 
ActivEEG"),  has  explored  tetherless  and  wireless  approaches.  Also,  the  GSR  Temp  2X  (Thought 
Technology  Ltd:  Montreal,  Canada)  offers  a  model  where  the  user  only  has  to  place  his  or  her 
fingers  lightly  on  top  of  the  device,  versus  being  strapped  to  it.  Nevertheless,  the  device  is  still 
attached  to  computer. 

5.6  Results  of  Literature  Search  on  Psychophysiological 
Measures 

The  literature  search  on  psychophysiological  measures  yielded  literature  containing 
psychophysiological  measures  that  either  directly  involved,  or  was  considered  useful  for 
implementation  into,  intelligent  tutoring  systems.  Table  7  presents  a  list  of  the  references 
examined  in  this  literature  review,  classified  by  type  of  psychophysiological  measure  (i.e.,  EEG, 
HRV,  or  GSR). 
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Table  7.  Psychophysiological  measures  literature  search:  Reference  list 


Type 

References 

EEG 

•  Thought  Technology  Ltd.  (2005) 

•  B -Alert 

•  St.  John,  Kobus,  &  Morrison  (2004) 

•  Chen,  &  Vertegaal  (2004) 

•  Blanchard  et  al.  (2007) 

HRV 

•  McCraty  (2002a,  2002b,  2003,  2009) 

•  Tiller,  McCraty,  &  Atkinson  (1996) 

•  McCraty  et  al.  (1995) 

•  Biocomtech 

•  emWave 

GSR 

•  Mandryk,  Inkpen,  &  Calvert  (2006) 

•  Or  &  Duffy  (2007) 

•  GSR  Temp  2X  (Thought  Technology  Ltd.,  2010) 

A  summary  of  desired  qualities  for  the  psychophysiological  measures  is  presented  in  Table  8.  The 

recommendations  and  suggested  use  of  psychophysiological  measures  for  adaptive  learning  were 

based  on  the  following  requirements: 

•  COTS:  Is  this  product  currently  available  commercially,  off-the-shelf? 

•  Affordable:  Is  this  product  within  the  reasonable  budget  to  spend? 

•  Unobtrusive:  Is  this  product  invasive,  bulky,  or  in  any  way  uncomfortable  so  that  it  may 
affect  desire  to  wear?  Also,  is  it  difficult  to  setup  and  calibrate  alone? 

•  User  Support:  Does  the  product  offer  any  form  of  technical  support  or  training  after 
initial  purchase? 

•  Build  Own:  Does  the  product  offer  instructions  on  how  to  build  it  oneself? 

•  Leave  for  IEDD  course:  Can  this  product  remain  in  CFB  Gagetown  for  use  by  the  IEDD 
course  students  when  the  project  is  completed? 
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Table  8.  Summary  of  Requirements  Checklist  for  Psychophysiological  Measures 


Style 

Product 

COTS 

Affordable 

Unobtrusive 

User 

Support 

Build 

Own 

Leave  for 
IEDD 
Course 

EEG 

Flexcomp  Infiniti  & 
Biograph  (Thought 
Technology  Ltd: 
Montreal,  Canada), 

yes 

maybe 

No 

yes 

no 

No 

B-Alert  (Advanced 

Brain  Monitoring  Inc: 
Carlsbad,  CA) 

yes 

? 

No 

yes 

no 

No 

HRV 

Heart  Tracker  3.0 
(Biocom  Technologes, 
Poulsbo,  WA) 

yes 

yes 

yes 

yes 

no 

yes 

Biocom  HRM 
(Biocom  Technologes, 
Poulsbo,  WA) 

yes 

no 

yes 

yes 

no 

yes 

EmWave  (HeartMath: 
Boulder  Creek,  CA) 

yes 

yes 

yes 

yes 

no 

yes 

HRV  Live!  (Biocom 
Technologes,  Poulsbo, 
WA) 

yes 

yes 

Yes 

yes 

No 

maybe 

Heart  Wizard 
(Advanced  Wellness 
Solutions:  Poulsbo, 

WA) 

yes 

yes 

yes 

yes 

no 

Yes 

GSR 

HandWave 
(Strauss,  Reynolds,  & 
Picard,  2005) 
(Massachusetts 

Institute  of 

Technology: 

Cambridge,  MA) 

no 

yes 

yes 

yes 

yes 

yes 

Mikron  Scan 
(temperature) 

(Or  &  Duffy,  2007) 
(Mikron  Instruments: 

San  Marcos,  CA) 

yes 

yes 

yes 

yes 

no 

yes 

Brainquiry  (Brainquiry 
Diagnostics:  New  York, 
NY) 

yes 

no 

yes 

yes 

no 

no 

GSR  Temp  2X 
(Thought  Technology 
Ltd:  Montreal, 

Canada) 

Yes 

Yes 

Yes 

yes 

no 

yes 
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5.7  Recommendations  and  Suggested  Use 


Given  the  results  of  the  requirements  checklist  in  Table  8,  the  HRV  Live!  and  GSR  Temp  2X  are 
two  possible  products  producing  indices  of  workload  that  are  recommended  for  purchase  within 
the  context  of  the  IEDD  course.  These  were  chosen  above  all  other  products  and  types  of 
measures  for  the  following  reasons: 

First,  at  this  time,  the  technology  supporting  the  EEG  is  not  affordable  or  unobtrusive  enough  to 
justify  its  inclusion  in  the  ITS.  The  mechanisms  underlying  the  capture  of  the  EEG  require  a 
multitude  of  gelled  electrodes  to  be  placed  carefully  on  the  skull.  This  feat  would  be  extremely 
difficult  and  cumbersome  for  students  to  set  up  on  their  own,  and  as  such,  this  would  defeat  the 
purpose  of  having  a  self-sufficient  tutor. 

The  reviewed  products  for  HRV  differ  primarily  by  price  and  their  analysis  capabilities.  Heart 
Tracker,  Heart  Wizard,  HRV  Live!  and  emWave  use  either  ear  or  finger  clips,  and  the  Biocom 
HRM  and  HRV  Live!  are  also  available  wirelessly  with  Bluetooth  (Bluetooth  SIG:  Kirkland, 
WA).  For  PC  use,  all  five  can  be  considered  equally  non-invasive.  In  terms  of  price,  EmWave  is 
the  most  affordable  at  under  $50  per  unit,  while  at  the  top  of  the  price  scale,  the  Biocom  HRM 
and  HRV  Live!  cost  approximately  $2,000  each.  Overall,  the  least  expensive  products  may  be 
ideal  to  leave  for  the  course  if  multiple  sessions  are  required.  In  addition,  when  one 
considers  the  less-than-sturdy  design  of  these  small  electronic  products,  and  that  as  a  result,  they 
may  need  to  be  replaced  quite  often,  this  actually  increases  the  current,  relatively -low  calculated 
cost  of  the  EmWave. 

Akin  to  eye-trackers,  the  primary  concern  in  selecting  a  specific  HRV  product  is  the  type  and 
accessibility  of  the  data  output.  At  first  glance  the  Heart  Tracker  3.0  or  the  Heart  Wizard  appear 
to  be  ideal  products.  Each  of  these  Biocom  (Biocom  Technologies:  Poulsbo,  WA)  products  is 
inexpensive,  collects  non-invasive  HRV  measurements,  and  then  analyses  them  against  individual 
traits,  such  as  biological  age,  physical  fitness,  and  lung  capacity,  in  order  to  derive  an  overall 
health  and  stress  measure.  This  single  output  (e.g.,  ranging  from  of  1  -  100)  is  the  ideal  type  of 
simple  reading  that  is  needed  to  fuel  the  ITS.  However,  further  investigation  has  revealed  that 
these  programs  are  only  able  to  provide  snapshots  of  the  user’s  data  that  are  retrospectively 
analyzed  by  the  researcher.  The  purpose  of  the  ITS  is  to  adapt  to  the  user’s  stress  level  in  real¬ 
time,  hence  the  need  to  stream  real-time  data  into  the  system. 

Another  product  from  Biocom,  HRV  Live!,  resolves  this  issue.  HRV  Live!  is  designed  as  a 
professional  research  tool  rather  than  a  consumer-based  health  and  wellness  assessment  tool.  As 
such,  it  streams  raw  HRV  data  in  real-time,  after  a  5-minute  system  configuration  which 
establishes  the  user’s  HRV  resting  baseline.  The  raw  data  are  not  combined  with  other  physical 
health  indicators  (e.g.,  biological  age),  which  is  not  a  large  concern  within  the  sample  of  military 
operators  who  should  be  in  peak  physical  fitness.  Although  HRV  Live!  is  significantly  more 
expensive  than  other  HRV  products,  it  is  the  only  product  that  meets  the  essential  requirements  of 
the  ITS. 

The  other  issue  to  consider  that  is  similar  to  eye-trackers  is  the  ability  of  the  ITS  to  access  the  raw 
data  output  from  HRV  Live!,  and  interpret  it’s  meaning  relative  to  the  student’s  learning.  The 
idea  is  that  HRV  will  be  used  to  assess  arousal.  Depending  on  the  content  presented  within  the 
course,  the  student  will  be  expected  to  be  aroused  or  not.  For  example,  when  conducting  a  threat 
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assessment,  low  arousal  levels  could  prompt  the  tutor  to  give  hints  or  probes  that  a  threat  should 
be  detected.  Alternatively,  if  the  user  is  over  aroused  (e.g.,  frustrated)  when  he  or  she  should  not 
be  (e.g.,  when  learning  introductory  material),  the  tutor  could  change  the  style  of  feedback  to  one 
more  helpful. 

While  it  is  desirable  to  start  the  initial  design  of  the  ITS  with  a  minimal  amount  of  adaptive 
mechanisms  to  avoid  confounding  technological  complexity,  future  attempts  to  triangulate  data 
are  recommended  by  incorporating  a  device  that  measures  the  GSR.  Specifically,  for  using  skin 
conductance  to  infer  mental  workload,  the  GSR  Temp  2X  is  the  recommended  COTS  model, 
while  the  HandWave  is  recommended  as  a  self-constructed  alternative.  It  is  to  be  noted  that  the 
HandWave  is  a  university  engineered  prototype  developed  in-house,  which  likely  increases  the 
difficulty  of  reproduction.  These  two  products  are  the  only  technologies  that  are  available, 
affordable,  and  unobtrusive.  While  the  Mikron  Scan  also  fits  these  categories,  it  was  used  for 
measuring  facial  skin  temperature  and  not  the  GSR  (and  was  included  as  an  interesting  alternative 
to  the  GSR,  but  not  yet  supported  by  enough  other  research  to  be  considered  for  serious 
purchase). 

In  addition  to  specific  technology  purchase  suggestions,  a  number  of  general  use 
recommendations  are  offered  to  aid  the  ease  of  integration  of  psychophysiological  measures  into 
adaptive  learning.  The  following  section  provides  a  list  of  recommendations  and  suggested  uses 
for  the  implementation  of  psychophysiological  technologies  within  the  IEDD  course  ITS  as  based 
on  the  literature  reviewed: 

Defining  the  Need:  The  context  in  which  the  ITS  will  be  used  will  be  a  determining  factor  in  the 
choice  of  a  psychophysiological  measure.  For  instance,  while  it  may  be  theoretically  ideal  to 
capture  multiple  forms  of  mental  workload  output,  it  may  be  equally  undesirable  to  attempt  to 
analyze,  interpret,  and  integrate  these  data  into  one  ITS. 

•  The  reviewed  research  suggests  that  some  workload  measures  are  related 
(e.g.,  HRV  and  the  EEG),  Thus,  it  may  be  more  feasible  to  use  a  single  measure  to 
infer  workload. 

•  The  issue  with  a  more  pragmatic  approach  to  using  workload  metrics  is  that  some 
measures,  such  as  the  GSR,  may  only  indicate  a  broad  presence  of  emotion,  but  not 
allow  for  further  detail.  It  is  likely  then,  that  emotions  such  as  frustration  could  be 
confused  with  arousal,  which  would  have  opposing  responses  from  an  ITS. 

•  In  addition,  the  physical  equipment  needed  to  use  multiple  indices  of  mental 
workload  will  be  limited  by  the  physical  constraints  of  the  actual  operators.  If  the 
ITS  is  intended  to  provide  additional  practice  training  that  operators  can  do 
independently  on  their  personal  laptops,  then  it  would  be  undesirable  to  have 
multiple  psychophysiological  monitors,  as  it  would  be  very  burdensome  to  set  these 
up  alone. 

o  On  the  other  hand,  if  the  tutoring  program  is  designed  exclusively  for  use  on  one 
computer  system  in  each  military  base,  which  could  be  technically  supported  for 
setup  and  calibration,  then  it  may  be  more  desirable  and  feasible  to  include  more 
complex  measures  of  cognitive  functioning. 
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•  The  other  option  is  to  start  with  an  uncomplicated  measure  such  as  HRV  measured 
by  a  commercial  product  such  as  EmWave.  Given  that  the  EmWave  is  small, 
inexpensive,  unobtrusive,  and  offers  wireless  options,  it  could  provide  a  “starting” 
point  for  future  investigations  of  adaptive  learning  systems  as  proposed  by 
Karamouzis  (2006). 

•  In  the  context  of  the  IEDD  course,  an  obvious  source  of  threat:  the  initial  detection 
of  the  IED,  followed  by  the  determination  of  level  of  threat  (i.e..  How  much  damage 
will  the  IED  cause  if  detonated?  How  likely  is  it  to  be  detonated?  Are  there  snipers 
in  the  area?  ).  In  order  to  improve  operator  questioning  techniques,  the  GSR  could 
be  paired  with  HRV  when  operators  are  interviewing  on-scene  commanders  or  other 
witnesses.  The  expectation  would  be  that,  when  pertinent  IED  information  is 
revealed,  the  student  would  show  signs  of  increased  arousal.  If  not,  the  tutor  would 
infer  that  the  student  missed  these  clues,  and  thus  could  offer  further  hints  and 
probes  to  guide  his  or  her  attention  to  this  important  information. 

Assess  Benchmark  Data:  EEG  research  is  only  recommended  for  very  specific  tasks  when  the 
EEG  results  are  known  ahead  of  time  (i.e.,  bench-markers)  and  can  be  compared  in  order  to  adapt 
the  system  appropriately. 

•  As  with  any  test,  individual  responses  will  vary  in  response  to  stimuli,  therefore, 
baseline  data  will  need  to  be  established  prior  to  testing  so  that  the  tutor  can  assess 
the  change.  Knowing  only  the  current  output  would  only  be  acceptable  for 
something  like  eye-tracking,  not  mental  workload  ,  which  varies  over  time. 

•  A  further  caveat  to  consider  is  the  many  artefacts  that  can  produce  variability  within 
these  psychophysiological  measures,  apart  from  just  workload  or  emotion  assigned 
to  the  task  at  hand.  For  example,  caffeine  and  sugar  intake  are  well  known 
stimulants  that  affect  arousal,  heart  rate,  etc.,  and  these  substances  are  likely 
consumed  by  many  potential  operators.  It  may  be  unrealistic  to  expect  students  who 
are  already  experiencing  stress  due  to  the  IEDD  course  exam  to  refrain  from  their 
daily  rituals.  This  is  so  because  refraining  from  these  rituals,  in  fact,  could  further 
distort  their  results  due  to  withdrawal  symptoms. 

Short  versus  long-term  observation:  It  can  be  recommended  that  when  defining  the  type  and 
needs  of  the  cognition  monitoring  technology  used  to  adapt  the  system,  the  time  span  in  which 
observation  is  required  should  be  defined.  This  recommendation  is  based  on  findings  that  certain 
types  of  psychophysiological  monitoring  technology  are  better  suited  for  long-  versus  short-term 
observation.  For  example,  research  regarding  skin  temperature  revealed  that  this  technology  was 
inappropriate  for  quasi,  real-time  adaptation  but  that  it  appears  to  be  suitable  for  longer-term 
observation  and  adaptivity  (Blanchard,  et  al.,  2007). 

Multimodality: 

•  Use  multiple  psychophysiological  measures:  It  is  recommended  that 
psychophysiological  measures  should  not  be  implemented  as  a  standalone  tool.  In 
other  words,  it  is  recommended  that  more  than  one  cognition  monitoring  technology 
system  should  be  used  in  order  to  “triangulate”  the  measures  of  interest.  As 
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suggested  by  Blanchard  et  al.  (2007),  in  order  for  the  system  to  be  designed  as 
relevant  for  a  learning  activity,  a  particular  pattern  detected  in  a  given  physiological 
channel  must  be  corroborated  with  other  sources  in  order  to  confirm  its  meaning. 
Complementary  data  should  come  from  other  physiological  signals. 

•  On  that  same  note,  an  additional  problem  with  multimodal  measurement  is  the 
multi-fold  data  output  that  comes  with  it.  For  example,  the  EEG  produces  a  vast 
amount  of  data  points,  which  must  then  be  interpreted  meticulously  by  the 
researcher.  Given  that  research  by  McCraty  et  al.  (1995)  has  successfully  linked  the 
EEG  to  heart  rate  measures  such  as  F1RV,  it  may  be  appropriate  to  use  less  complex 
methods.  Even  if  evidence  supports  EEG  measurement  accuracy,  the 
likelihood  of  its  use  decreases  dramatically  if  the  process  of  acquiring  these  data  is 
overly  complicated. 

Experts  Required: 

•  As  every  measure  is  quite  specific  in  terms  of  implementation,  and  the  collection 
and  analysis  of  associated  data,  it  is  recommended  that  an  expert  should  be 
associated  with  the  specific  measures  involved  in  ITS  design. 

•  The  current  project  may  warrant  the  use  of  individuals  who  have  expertise  pertaining 
to  a  specific  psychophysiological  measure.  This  is  because  the  complex  data  output 
from  a  particular  psychophysiological  measures  may  not  be  properly  interpreted  by 
the  average  researcher. 

•  Furthermore,  given  the  multitude  of  confounding  parameters  likely  to  exist  in  any 
given  measure  alone  (e.g.,  noise,  individual  differences,  etc.),  it  is  strongly 
recommended  that  the  proposed  ITS  be  tested  with  each  psychophysiological 
measure  separately  at  the  start  of  the  design. 

•  Within  the  constrained  environment  of  the  actual  IED  course,  a  concerning 
limitation  is  the  inability  to  test  between  subjects,  as  no  CF  student  is  to  be  denied 
the  best  training  available.  As  such,  it  is  recommended  to  test  the  effectiveness  of 
individual  mechanisms  on  civilians  who  are  not  employed  as  CF  IEDD  operators. 
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6  Performance  Tracking 


Within  performance  tracking  systems,  the  idea  is  to  record  the  student’s  progress  so  that  the 
system  adapts  to  changing  levels  of  the  student’s  performance  as  learning  increases.  One  note 
with  this  area  of  focus  is  that  there  is  a  lack  of  COTS  technology  available  for  adaptive  purposes. 
Due  to  this  limitation,  this  section  includes  mostly  conceptual  research  for  the  purpose  of 
recommending  ideas  for  future  use,  as  most  technologies  or  software  used  are  not  available  for 
use  in  the  current  project.  While  the  review  of  performance  tracking  systems  did  reveal  several 
areas  of  interest  for  developing  technology  (e.g.,  facial  and  motion  tracking),  the  bulk  of 
the  more  mature  (and  available)  literature  is  housed  in  eye-tracking,  speech  recognition  and 
language  training.  This  section  presents  the  results  of  the  detailed  review  of  attention  and 
performance  tracking. 

6.1  Eye-Tracking 

The  collection  of  users’  actions  is  not  complete  without  an  understanding  of  which  interface 
features  are  being  attended  (Bednarik,  2005).  Eye-movement  tracking  provides  instant 
information  about  the  location  of  the  user’s  visual  attention.  Assuming  that  the  object  under  the 
user’s  visual  attention  is  also  located  on  the  top  of  his/her  cognitive  processing  “stack,” 
knowledge  about  one’s  gaze  can  help  to  understand  which  features  of  an  interface  were  of  interest 
to  the  observer,  and  in  what  order  and  how  long  the  user  attended  to  each  feature.  As  eye-tracking 
research  and  technology  has  been  reviewed  extensively  in  Sub-section  2.1  of  this  report,  the 
current  section  will  focus  on  alternative  forms  of  performance  tracking  in  order  to  avoid 
redundant  content. 

6.2  Attention  Tracking 

Attention  can  be  regarded  as  the  gateway  that  allows  information  into  the  brain,  as  it  is  what  one 
chooses  to  focus  on.  The  human  mind  is  bombarded  with  millions  of  stimuli  and  it  must  have  a 
way  of  deciding  which  of  this  information  it  wishes  to  process.  Attention  is  generally  considered 
to  have  limited  capacity,  as  we  cannot  effectively  monitor  all  of  the  events  in  our  environments  at 
once,  or  for  extended  periods  of  time.  With  selective  attention,  one  chooses  to  allocate  more 
mental  resources  to  a  particular  stimulus,  while  limiting  others  (e.g.,  holding  a  conversation  in  a 
loud  party).  In  divided  attention,  one  divides  mental  resources  among  multiple  stimuli,  which 
ultimately  decreases  the  overall  availability  of  attention  for  each  added  task  (e.g.,  eating,  while 
driving,  while  using  a  cell  phone). 

As  the  focus  of  this  report  is  on  the  IEDD  ITS,  it  is  beyond  the  scope  of  this  review  to  give  a 
detailed  review  of  the  cognitive  factors  involved  in  attention.  The  general  idea  behind  attention 
tracking  in  the  context  of  adaptive  learning  is  that  knowing  how  and  what  the  student  is  attending 
will  allow  the  tutor  to  better  adjust  to  the  learner’s  needs.  One  form  of  attention  tracking  involves 
computer- mouse,  movement  tracking.  Here,  the  user’s  mouse  movements  on  a  computer  are 
considered  to  represent  the  information  that  is  being  attended  to,  which  is  similar  to  eye-tracking. 
With  computer  mouse  tracking,  the  tutor  can  assess  which  portions  of  a  webpage,  for  example, 
are  the  most  explored  by  the  user,  and  which  sections  remain  untouched. 
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Mcquiggan,  Mott,  and  Lester  (2008)  used  mouse  tracking  to  record  the  areas  that  participants  had 
explored  on  a  virtual  learning  island  in  order  to  determine  whether  information  should  be 
considered  new  or  learned.  The  online  tutorial  system  monitored  the  amount  of  time  spent  on 
each  question  and  how  long  the  cursor  resided  in  particular  locations.  The  interactive 
environment  then  tracked  the  time  elapsed  since  the  student  arrived  at  the  last  location,  achieved  a 
goal,  and  was  presented  with  an  opportunity  to  achieve  a  goal. 

Another  form  of  attention  tracking  is  posture  tracking.  For  example,  Dragon  et  al.  (2008) 
analyzed  students’  body  positions  and  seated  pressure  in  order  to  assess  their  level  of  engagement 
while  interacting  with  an  e-learning  environment.  These  measurements  were  then  correlated  with 
other  measures  such  as  the  GSR  to  infer  how  interested,  frustrated  or  bored  were  the  students. 

6.3  Motion  Tracking 

The  process  of  motion  tracking  can  also  be  categorized  as  a  form  of  attention  tracking,  but  on  a 
larger  scale.  Instead  of  analyzing  discrete  eye-movements  or  cursor  tracking  to  infer  what  an 
individual  is  paying  attention  to,  motion  tracking  captures  full-body  movement  data  in  order  to 
infer  various  aspects  of  cognition  as  based  on  behaviour.  For  example,  Livak  (2004)  used  motion 
tracking  to  assess  team  building  clearing  manoeuvres  as  based  on  stacking,  crouching,  and 
crawling  behaviour  on  objects  within  the  virtual  environment.  Sukthankar  and  Sycara  (2005)  also 
tracked  human  subjects  performing  various  movements  from  a  set  of  military  manoeuvres.  More 
specifically,  the  researchers  tracked  the  subjects’  motion  types  and  their  proximity  to  landmarks. 

The  transmission  of  realistic  human  motion  to  synthetic  agents  relating  to  a  specific  context  is 
well-suited  for  e-learning  training  environments.  Understanding  the  meaning  behind  physical 
behaviours  could  be  an  important  factor  to  consider  when  designing  the  non-verbal  aspects  of  the 
questioning  technique  for  the  IEDD  course.  For  the  purpose  of  this  course,  it  may  not  be  the  large 
body  movements  like  crouching  and  crawling  that  are  important,  so  much  as  the  hand  signals  and 
head  nods  that  occur  during  conversation.  On  the  other  hand,  full  body  movements  would 
certainly  be  important  during  any  dismounted  activity  within  the  IEDD  process,  such  as 
establishing  a  cordon  or  evacuating  the  area.  Furthermore,  actions  such  as  nervous  shifting  or 
pacing  could  be  used  to  train  lie  detection,  which  could  prove  particularly  useful  in  potential 
gaming  platforms  [e.g.,  Virtual  Battle  Space  2  (VBS2™)  (Bohemia  Interactive  Studio:  Prague, 
Czech  Republic)]  where  facial  expression  manipulation  is  limited.  VBS2™  is  a  gaming  platform 
that  was  designed  for  government  agencies.  It  offers  realistic  battlefield  simulations  and 
the  ability  to  operate  land,  sea,  and  air  vehicles.  It  also  delivers  a  synthetic  environment 
for  the  practical  exercise  of  the  behaviours  and  skills  that  are  required  to  successfully  execute 
various  missions. 

6.4  Facial  Recognition 

Another  way  to  infer  cognition  from  explicit  human  behaviour  is  through  facial  recognition.  This 
review  does  not  cover  a  great  depth  of  literature  on  this  topic,  as  facial  recognition  is  not  a  mature 
enough  technology  to  consider  at  the  moment.  Flowever,  facial  recognition  is  one  way  to 
automatically  infer  a  student’s  emotion  during  learning,  as  based  on  complex  algorithms  that 
analyze  the  dimensions  of  the  human  face.  Whitehill,  Bartlett,  and  Movelan.  (2008)  found  that 
facial  recognition  can  be  used  to  adjust  the  rate  of  presentation  for  learning  content  based  on 
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emotions  such  as  frustration  and  boredom.  Furthermore,  Craig,  D’Mello,  Witherspoon,  and 
Graesser  (2008)  related  participants’  facial  expressions  to  subjective  affective  states  while 
interacting  with  an  intelligent  tutoring  system. 

6.5  Speech  Recognition 

Speech  recognition  systems  are  described  as  a  technology  that  is  able  to  recognize  unrestricted 
speech  using  natural  language  processing.  Research  in  this  area  has  also  focused  on  other  related 
phenomena.  For  example,  researchers  have  examined  involuntary  changes  in  the  rate,  pitch, 
intensity,  quality  and  articulation  of  speech.  All  of  these  factors  have  also  been  associated  with 
different  emotions  (Murray  and  Arnott,  1993). 

Speech  recognition  (also  known  as  automatic  speech  recognition  or  computer  speech  recognition) 
converts  spoken  words  to  text.  The  term  "voice  recognition"  is  sometimes  used  to  refer  to  speech 
recognition  where  the  recognition  system  is  trained  to  a  particular  speaker  -  as  is  the  case  for  most 
desktop  recognition  software  (e.g.  RosettaStone:  Arlington,  VA);  thus,  there  is  an  element  of 
speaker  recognition,  which  attempts  to  identify  the  person  speaking,  to  better  recognize  what  is 
being  said. 

Speech  recognition  differs  from  voice  recognition,  as  it  is  broader  in  scope  given  that  it  refers  to  a 
system’s  ability  to  recognize  almost  anyone’s  speech.  In  contrast,  voice  recognition  is  a  system 
that  is  trained  to  a  particular  user,  as  it  recognizes  that  user’s  speech  based  on  his/her  unique 
vocal  sounds. 

6.6  Results  of  Literature  Search  on  Performance  Tracking 

The  literature  search  on  performance  tracking  technologies  yielded  several  sources  containing 
performance  or  attention  tracking  technologies  that  were  considered  relevant  with  respect  to  the 
current  intelligent  tutoring  system.  Table  9  presents  a  list  of  the  references  examined  in  this 
literature  review,  classified  by  the  type  of  attention  tracking  method  examined  (i.e.,  attention 
tracking,  performance  tracking  speech  recognition,  and  motion/face  recognition). 


Table  9.  Performance  tracking  literature  search:  Reference  list 


Type 

Reference 

Attention  Tracking 

• 

Van  Gog,  Jardozka,  Scheiter,  Gerjets,  &  Paas  (2009) 

• 

Jarmasz,  Keillor,  &  Hollands  (2005) 

Performance  Tracking 

• 

Sukthankar  &  Sycara  (2005) 

• 

Livak  (2004) 

Speech  Recognition 

• 

Kerly,  Ellis,  &  Bull  (2008) 

• 

El  Saadawi  et  al.  (2008) 

• 

RosettaStone 

• 

Johnson  (2009) 

• 

Johnson  &  Valente  (2008) 

Motion/Face  Recognition 

• 

Craig  et  al.  (2008) 

• 

Whitehill  et  al.  (2008) 
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In  addition,  a  summary  of  the  desired  qualities  for  performance  and  attention  tracking  are 
presented  in  Table  10.  This  table  was  based  on  a  list  of  requirements  designed  to  aid  in  the 
selection  of  the  most  appropriate  product  to  implement,  followed  by  recommendations  and 
suggested  uses  for  the  implementation  of  adaptive  learning  of  the  e-learning  framework  of  the 
IEDD  course,  as  based  on  the  literature  reviewed.  The  technologies  within  the  performance  and 
attention  tracking  literature  were  evaluated  based  on  the  following  criteria: 

•  COTS:  Is  this  product  currently  available  commercially,  off-the-shelf? 

•  Affordable:  Is  this  product  within  the  reasonable  budget  to  spend? 

•  Unobtrusive:  Is  this  product  invasive,  bulky,  or  in  any  way  uncomfortable  so  that  it  may 
affect  desire  to  wear?  Also,  is  it  difficult  to  setup  and  calibrate  alone? 

•  User  Support:  Does  the  product  offer  any  form  of  technical  support  or  training  after 
initial  purchase? 

•  Build  Own:  Does  the  product  offer  instructions  on  how  to  build  it  oneself? 

•  Leave  for  IEDD  course:  Can  this  product  remain  in  CFB  Gagetown  for  use  by  the  IEDD 
course  students  when  the  project  is  completed? 

•  Quality:  Has  this  product  been  supported  by  scientific  research  (i.e.,  high  quality)?  If 
not,  does  it  appear  to  be  usable  by  the  medical/scientific  community,  or  is  it  a 
commercial-only  product  without  any  form  of  validation  (i.e.,  low  quality)? 
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Table  10.  Summary  of  Requirements  Checklist  for  Attention  Tracking  Mechanisms 


Style 

Product 

COTS 

Affordable 

Unobtrusive 

User 

Support 

Build 

Own 

Leave  for 
IEDD 
Course 

Attention 

Tracking 

Tobii  1750 
(Tobii 

Technology: 

Sweden) 

Yes 

No 

Yes 

No 

No 

Yes 

Performance 

Tracking 

Wayang  Tutor 
(Wayang 

Outpost: 

Amherst,  MA) 

Yes 

Yes 

N/A 

N/A 

No 

No 

Crystal  Island 
with  Self  arch 
(NC  State 
University: 
Raleigh,  NC) 

No 

N/A 

N/A 

No 

Yes 

No 

CALMsystem 
(University  of 
Birmingham: 
Birmingham, 

UK) 

No 

N/A 

N/A 

No 

Yes 

No 

Report  Tutor 
(University  of 
Pittsburgh 

School  of 
Medicine: 
Pittsburgh  PA) 

No 

N/A 

N/A 

No 

Yes 

No 

RosettaStone 
(Arlington,  VA) 

Yes 

Yes 

N/A 

No 

Yes 

Yes 

Speech 

Recognition 

Alelo™, 

Tactical  Dari 
(Alelo:  Los 
Angeles,  CA) 

Yes 

Yes 

Yes 

maybe 

Yes 

Yes 

Motion/Face 

Recognition 

MOUT,  Unreal 

Tournament 

(Carnegie 

Mellon 

University: 
Pittsburgh,  PA) 

No/Yes 

Yes 

No 

No 

Yes 

No/Yes 

AutoTutor 
(University  of 
Memphis: 
Memphis,  TN) 

No 

N/A 

No 

No 

Yes 

No 
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6.7  Recommendations  and  Suggested  Use 


Ultimately,  the  review  of  performance  measures  and  attention  tracking  was  lacking  in  COTS 
products  that  realistically  could  be  implemented  in  the  IEDD  ITS.  The  most  applicable  research 
stemmed  from  AutoTutor,  which  described  a  program  to  help  medical  students  identify  patterns 
in  X-ray  scans,  however,  the  scan  and  zoom  process  used  in  this  program  is  not  directly 
transferable  to  the  IEDD  course  in  terms  of  product,  only  conceptually  so.  Other  programs,  such 
as  the  modified  Unreal  Tournament  offer  interesting  ideas  for  using  motion  tracking  as  an 
objective  and  non-invasive  method  of  tracking  performance.  However,  these  ideas  are  also 
academic  in  nature  and  are  not  easily  accessible  for  this  project.  Equally,  the  RosettaStone 
training  program  tracks  performance,  tailors  programs  to  the  user,  and  specifically  trains  new 
languages.  However,  RosettaStone  does  not  use  open-source  code  that  could  be  readily  adapted 
by  the  proposed  IEDD  ITS. 

Alelo™  presents  an  interesting  platform  to  incorporate  culture-based  training  to  the  IEDD 
operators.  Alelo™’s  Tactical  Language  and  Culture  Training  System  (TLCTS)  uses  cultural 
analysis  and  instructional  design  (Situated  Culture  Methodology)  to  facilitate  the  creation  of 
training  courses,  focusing  on  sociocultural  and  sociolinguistic  data  collection  on  situation  that 
trainees  are  most  likely  to  encounter  in  the  context  of  their  job  or  mission  context.  It  supports 
knowledge  and  skill  acquisition,  and  skill  transfer.  Alelo™  products  are  available  as  stand-alone 
courses,  embedded  in  other  courses  and  simulations,  or  blended  with  other  instruction 
modalities.  Every  course  is  specifically  designed  to  match  the  profile  of  its  target  students, 
accounting  for  factors  like  their  age,  native  language,  cultural  background,  native  country,  prior 
knowledge  of  the  target  language,  desired  level  of  fluency,  and  application  (business, 
professional,  tourism,  etc.). 

The  IEDD  Operator  Course  represents  an  ideal  application  for  speech  recognition  systems  like 
Alelo™.  Currently,  the  course  is  designed  to  train  the  Number  1  individual  (leader),  while  the 
Number  2  (assistant)  must  be  a  role  played  person  overly  qualified  for  this  position  (i.e.,  other 
Number  1  trainees  or  course  instructors).  Hence,  the  Number  1  team  position  requires  a  person  to 
act  as  a  "pseudo-assistant",  engaging  in  a  voice  dialog,  which  simulates  the  dialog  which  the 
Number  1  would  have  to  conduct  with  the  Number  2  individual  in  real  IEDD  situations.  Speech 
recognition  and  synthesis  techniques  offer  the  potential  to  eliminate  the  need  for  a  person  to  act  as 
pseudo-assistant,  thus  reducing  training  and  support  personnel 

Speech  recognition  systems  enable  the  intelligent  tutor  to  detect  and  correct  speakers’  dysfluency 
and  pronunciation,  and  to  provide  feedback  to  improve  the  speakers’  interpersonal  and 
communication  skills.  This  application  would  be  helpful  within  the  proposed  ITS  design  with 
respect  to  questioning  technique,  as  questioning  involves  interviewing  civilian  witnesses  from 
diverse  backgrounds,  which  requires  the  ability  to  converse  in  foreign  languages. 

Sensitivity  to  diverse  cultures  and  languages  is  surely  needed  when  conducting  interviews  of  high 
importance  regarding  IED’s.  However,  similar  to  RosettaStone,  the  limited  access  of  COTS 
systems  are  an  obstacle  to  using  existing  speech  recognition  systems  within  the  IEDD  course  as 
opposed  to  constructing  a  new  system  from  basic  components.  In  brief,  the  following  is  a  list  of 
key  issues  and  recommendations  regarding  the  implementation  of  performance  tracking  measures 
as  based  on  the  current  review: 
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•  The  majority  of  technologies  are  not  available  COTS  systems.  Those  that  are,  are  only 
gaming  platforms;  any  intelligent  portions  are  developed  in-house  [e.g.,  use  of  Military 
Operations  in  Urban  Terrain  (MOUT)]. 

•  Alelo™  appears  to  be  most  promising  because  it  already  has  developed  training  programs  for 
military  culture  training.  However,  it  is  unclear  whether  or  not  those  programs  could  be 
integrated  into  the  IEDD  ITS.  It  may  be  that  Alelo™  can  be  used  for  inspiration  only,  while 
the  proposed  ITS  will  be  built  using  and  authoring  tool  compatible  with  VBS2™. 

•  Speech  integration  would  be  ideal,  but  this  technology  is  not  mature  enough  at  this  time. 

o  An  alternative  to  using  actual  speech  recognition  may  be  to  use  text-based  selections 
that  are  pre-determined  in  the  system,  but  that  are  only  read  out  loud  when  selected 
by  the  user.  That  way,  it  would  be  more  realistic  to  hear  the  specific  intonations  and 
inflexions  in  the  speech  that  are  involved  in  different  types  of  questions,  instead  of 
only  being  able  to  read  the  phrase  and  infer  these  qualities  in  one’s  head. 

o  Perhaps  the  interviewer  (i.e.,  student)  could  “try  the  question  out”  before  selecting  it. 
For  instance,  the  interviewer  could  scroll  over  each  sentence  to  hear  what  it  would 
actually  sound  like  before  deciding  to  vocalize  the  sentence  to  the  interviewee. 

In  summary,  the  only  concrete  recommendations  that  can  be  made  at  this  time  are  to  further 
investigate  Alelo™  to  determine  the  accessibility  of  the  architectures  that  they  have  authored 
within  the  VBS2™  gaming  system. 
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7  Recommendations  and  Conclusions 


This  report  reviewed  current  techniques  and  technologies  relating  to  eye-tracking,  physiological 
indices  of  workload  and  stress,  cognitive  learning  styles  and  performance  tracking  that  have  been 
successfully  used,  or  could  theoretically  be  used  to  augment  an  ITS.  The  purpose  of  the  review 
was  to  provide  straightforward,  applicable  recommendations  that  would  lead  to  the  purchase  of 
technology  that  can  be  implemented  into  the  IEDD  ITS. 

Eye-tracking  as  a  means  of  attention,  workload,  and  expertise  measurement  proved  to  be  the  most 
mature  and  commercially  accessible  technology  used  in  research.  The  recommended  style  of  eye- 
tracker  is  a  desktop  or  screen-integrated  eye-tracker,  as  these  are  the  most  non-invasive  types  of 
eye-trackers,  and  they  take  the  least  time  to  set  up  and  calibrate  as  compared  to  head-mounted 
eye-trackers.  Goggle  style  eye-trackers  are  appealing  for  their  mobility,  but  are  unnecessary  and 
costly  for  use  in  front  of  a  PC.  The  main  limiting  factor  is  price,  as  the  high  quality  models  that 
are  frequently  used  in  research  tend  to  cost  over  $40,000.  The  suggested  quality/price 
compromise  is  to  purchase  one  mid-range  ($10,000  or  less)  eye-tracker  (such  as  the  EasyGaze 
system)  for  use  in  the  collection  of  baseline  data  and  the  calibration  of  other  systems.  If  numerous 
eye-trackers  are  needed  because  they  are  to  be  left  at  the  IEDD  course,  it  is  recommended  that 
open-source,  or  web-cam-based  alternatives  be  used,  as  these  can  be  calibrated  against  a  mid¬ 
range  eye-tracker  that  is  also  recommended  for  purchase. 

It  is  recommended  that  the  ILS  (Felder  &  Solomon,  2006)  be  administered  to  students  prior  to 
their  use  of  the  IEDD  ITS.  The  ILS  has  proven  validity  and  reliability,  is  freely  available  and 
automatically  scored  online,  and  takes  only  a  few  minutes  to  complete.  It  is  further  recommended 
that  the  ILS  be  administered  during  baseline  testing  in  order  to  get  a  sense  of  any  dominant 
learning  styles.  These  results  would  help  provide  guidance  for  content  building  within  the  actual 
ITS  (e.g.,  if  there  was  a  preference  for  visual  over  auditory  learning  styles  among  students). 

The  EEG,  while  a  valuable  and  proven  source  of  information  about  mental  workload  and  other 
cognitive  functions,  is  not  recommended  as  a  form  of  adaptive  measure  within  the  IEDD  ITS. 
First  and  foremost,  the  equipment  and  set-up  time  required  to  acquire  accurate  EEG  recordings 
makes  it  unlikely  to  be  usable  by  IEDD  operators  without  assistant  or  researcher  supervision.  The 
IEDD  ITS  needs  to  be  a  stand-alone  learning  companion  that  is  readily  accessible  to  students.  If 
the  IEDD  ITS  requires  an  hour  for  placing  gels  and  electrodes  on  one’s  scalp,  it  is  doubtful  that  it 
will  be  used  beyond  pure  research  puiposes  when  the  experimenter  is  present. 

The  psychophysiological  measures  recommended  for  use  are  the  HRV  and  the  GSR  in  either  a 
simple  ear-clip,  or  via  a  wireless  Bluetooth  model.  Both  measures  have  feasible,  non-invasive 
techniques  for  assessment,  and  could  be  used  in  combination  to  help  triangulate  a  more 
accurate  meaning  of  general  arousal  to  specific  stimuli.  The  primary  technology 
recommended  for  purchase  is  the  HRV  Live!,  while  the  GSR  Temp  2x  is  recommended  for  future 
data  triangulation. 

Performance  tracking  measures  produced  the  smallest  number  of  tangible  technologies  that  could 
be  implemented  into  the  IEDD  ITS.  While  motion  tracking  offered  some  interesting  research 
streams  for  performance  tracking  as  based  on  posture,  at  this  point,  the  related  technologies  are 
only  academic  prototypes.  The  most  plausible  form  of  performance  tracking  appears  to  be  speech 
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recognition.  This  has  matured  in  the  field  of  language  training  as  in  RosettaStone,  but  is  still  in 
need  of  improvement  for  Alelo™.  The  problem  is  that  the  project  is  not  in  need  of  a  language 
training  just  yet,  so  this  technology  is  likely  too  complex  and  not  reliable  enough  to  use  yet.  As 
for  Alelo™,  the  style  of  its  interactive  and  mobile  training  is  certainly  appealing  -  the  question  is: 
can  the  application  itself  can  be  integrated  into  the  IEDD  ITS? 

In  summary,  the  results  of  the  literature  review  pertaining  to  adaptation  mechanisms  demonstrate 
a  range  of  maturing  technologies  that  can  be  usefully  applied  to  the  development  of  an  intelligent 
tutoring  system  designed  to  augment  the  IEDD  Operator  Course.  In  brief,  the  products 
recommended  for  immediate  purchase  are: 

•  Eye-tracker:  EasyGaze  -  Design  Interactive. 

•  Psychophysiological:  HRV  live!  -  Biocom. 

•  Performance  Tracker/Mobile  Application:  Tactical  Dari  (tentative)  -  Alelo™. 

Once  these  technologies  are  acquired,  studies  need  to  be  carried  out  to  investigate  design  and 
implementation  approaches  for  integrating  these  technologies  with  the  IEDD  Operator  Course. 
An  ITS  evaluation  plan  also  needs  to  be  developed  to  assess  the  effectiveness  of  these  ITS 
adaptation  mechanisms  on  IEDD  training. 
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